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Abstract 

Military  and  civil  space  acquisitions  have  reeeived  mueh  eritieism  for  their 
inability  to  produee  realistie  eost  and  sehedule  estimates.  This  researeh  seeks  to  provide 
space  systems  cost  estimators  with  a  forecasting  tool  for  space  system  eost  and  schedule 
growth  by  identifying  faetors  eontributing  to  growth,  quantifying  the  relative  impaet  of 
these  faetors,  and  establishing  a  set  of  models  for  predieting  spaee  system  eost  and 
sehedule  growth.  The  analysis  eonsiders  data  from  both  Department  of  Defense  (DoD) 
and  National  Aeronauties  and  Space  Administration  (NASA)  spaee  programs. 

The  DoD  dataset  ineludes  21  spaee  programs  that  submitted  developmental 
Seleeted  Aequisition  Reports  between  1969  and  2006.  The  analysis  uses  multiple 
regression  to  assess  22  predictor  variables,  finding  that  eommunieations  missions,  ground 
equipment,  firm- fixed  priee  eontraets,  and  inereased  program  manager  tenure  are  all 
predietive  of  lower  cost  growth  for  military  space  systems. 

The  NASA  analysis  ineludes  71  satellites  and  spaeeeraft  developed  between  1964 
and  2004.  The  analysis  uses  a  two-stage  logistic  and  multiple  regression  approaeh  to 
analyze  3 1  predictor  variables  finding  that  smaller  programs  (by  total  eost),  more  massive 
spaeeeraft,  mierogravity  missions,  and  spaee  physies  missions  are  predietive  of  higher 
eost  growth.  For  sehedule  growth,  the  study  finds  that  larger  programs  and  those 
developed  by  the  Jet  Propulsion  Laboratory,  Northrop  Grumman,  or  international 
developers  are  predietive  of  inereased  sehedule  growth,  whereas  those  programs 
developed  by  Johns  Hopkins  University  are  predictive  of  reduced  sehedule  growth. 
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PREDICTING  COST  AND  SCHEDULE  GROWTH  FOR  MILITARY  AND  CIVIL 

SPACE  SYSTEMS 

I.  Introduction 

Over  the  past  deeade,  the  United  States  has  grown  increasingly  dependent  on 
space  systems  in  order  to  conduct  military  and  civil  operations.  The  combination  of  this 
dependence  and  the  recent  difficulties  in  space  systems  acquisition  has  given  cause  for 
alarm  among  national  leaders  (Allard,  2005;  Defense  Science,  2003).  Space  acquisition 
programs  such  as  the  Space  Based  Infrared  System  (SBIRS)  High  and  the  National  Polar- 
orbiting  Operational  Environmental  Satellite  System  (NPOESS)  have  received 
considerable  national  attention  (and  Congressional  criticism)  for  their  excessive  cost 
growth.  Eor  example,  SBIRS  High,  originally  estimated  to  cost  approximately  $4  billion, 
is  now  anticipated  to  cost  over  $10  billion.  Similarly,  NPOESS  has  almost  doubled  in 
cost  growth,  from  an  original  estimate  of  approximately  $6  billion  to  current  estimates  of 
over  $11  billion  (Defense,  2003a;  Government,  2006).  The  extreme  cost  growth 
experienced  by  these  and  other  military  and  civil  space  acquisition  programs  has  led  to 
the  perception  that  the  space  acquisition  process  is  “broken,”  ultimately  eroding  the 
credibility  of  the  space  acquisition  community  (Allard,  2005;  Gourley,  2004;  Eee,  2004). 

This  study  seeks  to  assist  cost  estimators  by  providing  the  military  and  civil  space 
systems  acquisition  communities  with  a  set  of  models  for  predicting  the  likelihood  and 
quantity  of  space  system  cost  and  schedule  growth.  These  models  will  enable  space 
system  cost  estimators  to  enhance  their  current  estimating  techniques  as  well  as  identify 
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the  primary  factors  associated  with  space  system  cost  and  schedule  growth.  Hopefully, 
the  creation  of  these  models  will  result  in  better  forecasting,  and  thus  decreased  future 
cost  and  schedule  growth,  in  the  acquisition  of  space  systems. 

This  chapter  provides  an  overview  of  this  study’s  efforts  to  understand  and  model 
space  system  cost  and  schedule  growth  by  examining  the  space  system  acquisition 
background,  the  specific  research  problem,  the  research  objectives,  and  the  methodology. 
The  chapter  concludes  with  an  overview  of  the  study  results. 

Background:  Space  System  Acquisition 

Military  space  system  acquisition  began  in  the  1950s  with  the  development  of 
ballistic  missiles  by  the  Western  Development  Division  of  the  Air  Research  and 
Development  Command  (ARDC).  In  1955,  ARDC  expanded  its  mission  by  taking  on  the 
responsibility  of  developing  the  Department  of  Defense’s  (DoD’s)  first  satellite.  Civil 
space  system  acquisition  began  in  1958  under  the  National  Aeronautics  and  Space 
Administration  (NASA)  as  a  response  to  the  Soviet  launch  of  Sputnik.  While  the  DoD 
focused  on  strategic  missile  development  and  defense  satellites,  NASA’s  role  included 
human  space  flight  and  scientific  space  exploration  (“Brief,”  2005).  Although  many  of 
the  missions  for  the  DoD  and  NASA  overlap  (such  as  space-based  communications, 
weather  observation,  and  environmental  monitoring),  NASA  does  not  fall  under  the 
purview  of  the  DoD,  rather  NASA  operates  as  an  Independent  Agency  (“Official,”  2008). 

In  1961,  Secretary  of  Defense  Robert  McNamara  gave  the  Air  Force  primary 
responsibility  for  developing  all  military  space  systems  (History,  2003).  In  2003,  the 
DoD  reaffirmed  the  Air  Force’s  role  in  developing  space  systems  when  Deputy  Secretary 
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of  Defense  Paul  Wolfowitz  designated  the  Seeretary  of  the  Air  Foree  as  the  DoD 
Exeeutive  Agent  for  Spaee^  (Department,  2003a). 

The  Air  Foree’s  Spaee  and  Missile  Systems  Center  (SMC)  is  the  eurrent-day 
sueeessor  to  the  Western  Development  Division  of  the  1950s  (History,  2003).  SMC  was 
originally  aligned  under  the  eontrol  of  Air  Foree  Materiel  Command  (AFMC),  the  Air 
Foree’s  primary  aequisition  arm.  However,  in  2001,  upon  the  reeommendation  of  the 
Spaee  Commission,  SMC  was  realigned  under  the  eontrol  of  Air  Foree  Spaee  Command 
(AFSPC).  The  Spaee  Commission  argues  that  plaeing  SMC  under  AFSPC  would 
eonsolidate  the  operational  and  aequisition  funetions  for  spaee  into  a  single  organization, 
thus  aehieving  a  “strong  eenter  of  advoeaey  for  spaee”  as  well  as  fostering  an 
organizational  elimate  suitable  for  developing  spaee  professionals  {Report,  2001:89-90). 

In  addition  to  having  a  separate  aequisition  eommunity  for  military  spaee  systems, 
there  is  also  a  separate  proeess.  While  typieal  DoD  aequisition  follows  the  proeess 
outlined  in  the  DoD  Instruetion  5000.1,  “The  Defense  Aequisition  System,”  and  5000.2, 
“Operation  of  the  Defense  Aequisition  System,”  military  spaee  system  aequisition 
follows  a  separate  aequisition  proeess  deseribed  in  National  Seeurity  Spaee  (NSS)  03-01. 
Through  NSS  03-01,  all  DoD  spaee  aequisition  programs  follow  a  separate  reporting 
ehain  from  other  DoD  programs  and  are  automatieally  granted  waivers  from  DoD 


*  The  National  Reconnaissance  Office  (NRO)  has  responsibility  for  the  development  of  reconnaissance 
satellites  (History,  2003:1).  This  does  not  diminish  the  Air  Force’s  responsibility  for  space  systems;  for, 
the  Under  Secretary  of  the  Air  Force  also  serves  as  the  Director  for  the  NRO  (Department,  2003a:3). 

^  The  Space  Commission  recommends  the  creation  of  a  Space  Corps  within  the  Air  Force  under  AFSPC  as 
a  mid  term  solution;  in  the  long  term,  it  recommends  a  separate  military  department  for  space  (Report, 
2001:89). 

^  Although  Deputy  Secretary  of  Defense  Paul  Wolfowitz  cancelled  the  DoD  5000  series  in  2002 
(Wolfowitz,  2002),  the  DoD  acquisition  community  continues  to  use  much  of  this  process  through  the 
discretionary  use  of  the  Defense  Acquisition  Guidebook,  which  details  implementation  of  the  DoD  5000 
series  (Vogel,  2003:4;  “Defense,”  2004). 
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Instruction  5000.2  (Department,  2003b;  Fritehman,  2005).  NSS  03-01  provides  spaee 
system  aequisition  professionals  with  a  flexible  and  streamlined  proeess  tailored  towards 
the  unique  aspeets  of  spaee  system  aequisition  (Department,  2003b). 

Military  and  eivil  spaee  systems  differ  from  other  defense  systems  in  two  ways: 
their  operational  environment  and  their  aequisition  life  cyele."^  The  operational 
environments  for  spaee  systems  are  harsh  and  remote.  Spaee  systems  have  to  deal  with 
extremes  (sueh  as  radiation,  eharged  partieles,  and  the  vaeuum  of  spaee)  that  land-based 
systems  do  not.  Additionally,  due  to  the  remote  nature  of  the  operational  environment,  it 
is  diffieult  to  make  eorreetions  or  modifieations  to  the  systems  onee  they  have  been 
deployed.  The  emphasis  plaeed  on  system  survivability  of  this  harsh  environment  results 
in  large  eosts  for  spaee  systems  during  the  early  stages  of  the  aequisition  life  eyele. 
Beeause  testing  in  the  operational  environment  (spaee)  is  unrealistie,  spaee  systems 
aequisition  plaees  a  stronger  emphasis  on  test  and  evaluation  during  the  development 
phase  (Fritehman,  2005;  Sellers  et  ah,  2004). 

In  addition  to  the  emphasis  plaeed  on  the  aetivities  of  the  development  phase, 
spaee  systems  also  differ  in  their  aequisition  life  eyele.  Spaee  systems  are  often  aequired 
in  small  quantities  and  usually  do  not  have  maintenanee  performed  on  them  onee  they 
beeome  operational.  Beeause  of  the  limited  quantities  and  the  high  development  eost  of 
spaee  systems,  their  aequisition  life  eyele  does  not  typieally  inelude  an  extensive 
produetion  phase  or  the  use  of  prototyping  (Fritehman,  2005).  As  ean  be  seen  in  Figure 
1,  a  typieal  defense  weapon  system  experienees  the  majority  of  its  life  eyele  eost  during 

The  acquisition  life  cycle  of  a  program  includes  all  of  the  phases  for  developing  and  producing  a  system 
from  the  initial  concept  through  operations  and  sustainment  of  the  system  (“DAU,”  2007). 
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the  operations  and  support  phase,  after  the  system  has  been  deployed.  Space  systems,  on 
the  other  hand,  experience  most  of  their  costs  during  the  system  acquisition  phase  (Figure 
2),  where  system  design,  integration,  and  testing  occurs. 


MS  MS  MS 

"A"  "B"  "C" 

Figure  1.  Typical  Weapon  System  Life  Cycle  Cost  Curve  (adapted  from  Paschall,  2005) 


KDP  KDP  KDP 

"A"  "B"  "C" 

Figure  2.  Typical  Space  System  Life  Cycle  Cost  Curve  (adapted  from  Paschall,  2005) 
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NSS  03-01  accommodates  the  unique  life  cyele  for  spaee  systems  by  offering  two 
aequisition  models:  the  NSS  Small  Quantity  System  Model  (Figure  3)  and  the  NSS  Large 
Quantity  Production  Focused  System  Model  (Figure  4).  The  NSS  Small  Quantity  System 
Model  is  designed  for  programs  that  typieally  aequire  ten  or  less  units,  sueh  as  satellites, 
ground  stations,  and  launeh  vehieles.  Distinetive  features  of  this  model  inelude  the 
Follow-on  Buy  Approval  and  the  Upgrade  Deeision  in  Phase  C  (Figure  3).  The  Follow- 
on  Buy  Approval  meeting  oecurs  after  the  first  or  seeond  unit  beeomes  operational. 
During  this  meeting,  the  deeision  is  made  as  to  whether  or  not  to  eomplete  the  small 
quantity  proeurement.  The  Upgrade  Deeision  meeting  provides  a  forum  to  approve  new 
requirements  that  oeeur  after  Key  Deeision  Point  C  (Department,  2003b).  The  NSS 
Large  Quantity  Produetion  Foeused  System  Model  applies  to  systems  that  are  typieally 
aequired  in  units  of  50  or  more.  Large  quantity  aequisitions  for  spaee  systems  are 
primarily  user  equipment,  sueh  as  hand-held  user  terminals.  The  NSS  Large  Quantity 
Model  is  similar  to  the  life  eyele  model  used  in  typical  DoD  acquisitions.  As  can  be  seen 
from  Figure  4,  this  model  ineludes  Low-Rate  Initial  Produetion  and  Full-Rate  Produetion 
in  Phase  C,  which  are  eommon  in  standard  DoD  aequisitions  (Department,  2003b). 
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Figure  4.  NSS  Large  Quantity  Production  Focused  System  Model  (adapted  from  Department,  2003b) 


The  Government  Aceountability  Office  (GAO)  notes  a  number  of  other 
differences  between  NSS  03-01  and  the  DoD  5000  series  including  the  use  of  a  Defense 
Space  Acquisition  Board  (DSAB)  early  in  the  acquisition  process,  an  emphasis  on  a 
revolutionary  acquisition  approach,  and  not  requiring  commitment  to  fully  fund  the 
program.  While  the  Air  Force  claims  that  the  use  of  DSABs  will  result  in  early  problem 
identification  and  enable  early  involvement  of  senior  leadership,  the  GAO  argues  that  the 
DSAB  will  result  in  investment  decisions  being  made  earlier  in  the  process,  before 
critical  technologies  are  mature  enough  to  enter  product  development.  The  GAO 
expresses  concern  about  NSS  03-0 1’s  practice  of  encouraging  programs  to  incorporate 
cutting-edge  technologies  through  a  revolutionary  approach;  it  argues  that  the 
simultaneous  development  of  product  and  technology  results  in  higher  risk  programs. 
Additionally,  NSS  03-01  does  not  require  a  commitment  to  fully  fund  the  program  at 
program  initiation  (Milestone  B),  whereas  other  DoD  acquisition  programs  are  required 
to  commit  to  full  funding  upon  initiation.  Rather,  NSS  03-01  gives  the  DoD  Space 
Milestone  Decision  Authority  the  flexibility  to  discontinue  funding  the  program  as  late  as 
the  Follow-on  Buy  Decision,  which  occurs  after  the  first  few  units  have  become 
operational.  The  GAO  argues  that  these  key  differences  between  NSS  03-01  and 
traditional  acquisition  policy  encourage  space  systems  acquisition  to  take  on  unnecessary 
technical,  schedule,  and  cost  risk  (Government,  2004a;  Department,  2003b).  The  GAO’s 
concerns  about  the  risks  in  space  acquisition  and  the  ramification  of  these  risks  are  not 
unfounded.  Numerous  cost,  schedule,  and  technical  problems  are  occurring  within 
current  space  system  acquisitions,  causing  Congress  and  the  American  public  to  believe 
that  the  space  system  acquisition  process  is  “broken”  (Tauscher,  2007;  Lee,  2004). 
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Concerns  over  cost  and  schedule  growth  are  not  limited  to  the  DoD;  the  GAO  has 
also  eriticized  NASA  for  its  inability  to  produce  realistie  eost  and  sehedule  estimates. 
While  NASA  points  to  teehnieal  problems  and  funding  shortages  as  major  eontributors  to 
cost  and  schedule  growth,  the  GAO  finds  that  the  problem  is  not  a  program  management 
issue,  but  rather  that  NASA  laeks  a  rigorous  proeess  for  aeeurately  estimating  eost  and 
sehedule.  NASA  eost  estimators  laek  access  to  sound  financial  and  technical  data,  and 
thus  are  unable  to  produee  reliable  estimates  (Government,  2004b). 

In  response  to  the  eritieisms  of  the  GAO  and  other  eritieisms,  NASA  recently 
revamped  its  proeedural  requirements.  The  revised  NASA  Proeedural  Requirements, 
NPR  7120.5D  standardizes  the  program  life-cyele  and  program  reviews,  as  well  as 
ineorporates  the  Key  Decision  Points  (KDPs)  found  in  the  defense  acquisition  life  cyele 
(Blythe,  2007).  Figure  5  displays  the  acquisition  life  eyele  found  in  NPR  7120.5D.  In 
many  ways,  the  proeess  is  similar  to  the  DoD’s  space  aequisition  process,  with  heavy 
emphasis  plaeed  on  the  upfront  development  activities  and  requiring  approval  at  each 
KDP  in  order  to  progress  to  the  next  phase.  Aceording  to  NPR  7120.5D,  “NASA  places 
signifieant  emphasis  on  projeet  formulation”  (National,  2007).  The  emphasis  on  early 
program  formulation  through  the  number  and  frequeney  of  teehnieal  and  programmatic 
reviews  appears  to  exeeed  the  reviews  outlined  by  NS S  03-01  for  defense  space  systems. 
During  Pre-Systems  Acquisition,  prior  to  program  implementation  at  KDP  C,  NASA 
spaee  systems  can  expect  to  go  through  four  program  reviews:  Mission  Concept  Review 
(MCR),  System  Requirements  Review  (SRR),  System  Definition  Review  (SDR),  and 
Preliminary  Design  Review  (PDR).  DoD  space  systems,  on  the  other  hand,  are  only 
subject  to  one  review,  SRR,  during  their  Pre-Systems  Aequisition  Phase. 
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Research  Problem:  Space  System  Cost  and  Schedule  Growth 

How  can  the  military  and  civil  space  acquisition  communities  correct  their 
problems  of  excessive  cost  and  schedule  growth?  One  critical  step  is  to  improve  cost  and 
schedule  estimates  for  space  systems  acquisition.  As  with  its  assessment  regarding 
NASA,  the  GAO  finds  that  DoD  space  cost  estimators  are  producing  unrealistic 
estimates.  In  the  DoD’s  case,  space  cost  estimators  have  a  tendency  to  make  unrealistic 
assumptions,  thus  creating  estimates  that  are  highly  optimistic.  This  underestimation  of 
program  costs  leads  to  cost  growth  as  the  programs  develop  (Government,  2006). 

What  is  wrong  with  optimistic  cost  and  schedule  estimates?  Underestimating 
program  costs  or  schedules  hinders  senior  leadership’s  ability  to  effectively  plan  and 
make  decisions.  When  a  program’s  cost  or  schedule  estimate  does  not  reflect  reality, 
financial  planners  are  unable  to  adequately  allocate  the  correct  budgetary  resources  for 
the  correct  time.  As  a  program  requires  more  funds  than  were  originally  planned, 
decision  makers  have  to  respond  by  either  reducing  the  quantity  or  performance  for  the 
system,  or  they  have  to  reallocate  funds  from  other  programs  in  order  to  make  up  for  the 
shortfall  (Arena  et  ah,  2006).  Reallocating  funds  from  other  programs  negatively  impacts 
these  programs’  ability  to  meet  their  respective  schedule  and  performance  requirements. 
Regardless  of  whether  decision  makers  choose  to  have  the  funding  shortfall  impact  the 
original  program  or  another  program,  the  end  result  is  that  the  delivered  mix  of 
capabilities  will  not  meet  those  that  were  originally  intended  by  strategic  planners.  A 
similar  problem  occurs  if  changes  to  the  schedule  result  in  funds  being  required  at  a 
different  time  than  originally  planned.  Additionally,  increases  in  schedule  affect  the 
capability  mix  by  not  delivering  systems  to  the  end  users  at  the  appropriate  time. 
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Another  consequence  of  cost  and  schedule  growth  is  that  it  decreases  the 

credibility  of  cost  estimators  and  the  space  acquisition  community,  which  can  ultimately 

hinder  space  acquisition  programs.  United  States  Senator  Wayne  Allard  (2005) 

expressed  this  sentiment  at  a  Space  Policy  and  Architecture  Symposium: 

The  Air  Force  and  its  contractors  have  lost  all  credibility  with  Congress  when  it 
comes  to  space  acquisition  programs.  My  colleagues  and  I  are  no  longer  surprised 
by  additional  cost  increases  or  notices  of  further  schedule  delays.  Nor  do  some  in 
Congress  give  much  credence  to  the  Air  Force’s  proposals  to  fix  these  programs. 
The  Congress’s  lack  of  confidence  in  Air  Force  space  acquisition  management 
has  resulted  in  enormous  reductions  in  funding  for  space  programs. 

Senator  Allard’s  comment  reveals  that  this  loss  of  credibility  has  already  resulted  in  a 

Congressional  response  of  reducing  funds  for  space  programs.  Thus,  in  addition  to 

funding  shortfalls  from  underestimating  costs,  the  space  acquisition  community  also  has 

to  struggle  with  funding  reductions  caused  by  their  loss  in  credibility. 

This  study  seeks  to  mitigate  these  impacts  of  cost  and  schedule  growth  by 

providing  space  systems  cost  estimators  with  a  forecasting  tool  for  space  system  cost  and 

schedule  growth.  In  so  doing,  this  study  identifies  factors  contributing  to  space  system 

cost  and  schedule  growth,  quantifies  the  relative  impact  of  these  factors,  and  provides  a 

set  of  models  for  predicting  space  system  cost  and  schedule  growth. 

Research  Objectives 

The  purpose  of  this  study  is  to  answer  the  question,  “Is  it  possible  to  create  a  set 
of  models  that  accurately  predict  the  likelihood  and  quantity  of  cost  and  schedule  growth 
for  space  systems?”  In  order  to  answer  this  research  question,  this  study  first  examines 
five  Investigative  Questions  (IQs): 
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IQl.  Which  systems  should  be  considered  “space  systems”? 

For  the  purpose  of  this  study,  whieh  systems  should  be  eonsidered  spaee  systems? 
As  discussed  herein,  military  space  system  acquisition  began  with  the  Western 
Development  Division,  which  was  responsible  for  acquiring  both  strategic  missiles  and 
satellites.  The  current  day  space  acquisition  arm  for  the  Air  Force,  SMC,  is  responsible 
for  acquiring  launch  vehicles  and  ground  equipment  in  addition  to  the  satellites  and  land- 
based  strategic  missiles  acquired  by  its  predecessor  (History,  2003).  Civil  space  system 
acquisition  on  the  other  hand,  focuses  primarily  on  satellites  and  spacecraft.  With 
regards  to  military  systems  should  “space  systems”  include  all  of  these  types  of  systems 
since  the  space  acquisition  community  holds  the  responsibility  for  acquiring  them?  Or 
should  “space  systems”  refer  only  to  those  that  fall  under  NSS  Small  Quantity  System 
Model,  which  are  similar  to  civil  space  systems? 

This  study  chooses  to  include  strategic  missiles,  launch  vehicles,  and  ground 
equipment  in  the  definition  of  space  system.  However,  recognizing  that  strategic  missiles 
are  not  being  widely  acquired  today  and  that  these  systems  do  not  fall  under  NSS  03-01, 
the  study  also  opts  to  provide  models  that  exclude  these  systems.  See  Chapter  II  for 
further  information  on  space  system  definitions. 

IQ2.  What  are  the  current  methods  for  calculating  cost  and  schedule  growth? 

By  examining  the  current  methods  for  calculating  cost  and  schedule  growth,  this 
study  can  select  an  appropriate  method  for  calculating  growth.  For  cost  growth,  this 
investigative  question  includes  the  determination  of  which  aspects  of  cost  growth  are 
important.  That  is,  do  all  increases  in  cost  qualify  as  cost  growth?  Perhaps  some 
increases  in  cost  do  not  reflect  a  true  increase  in  the  costs  estimated  for  the  original 
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program.  For  example,  an  inerease  in  eost  due  to  a  ehange  in  the  number  of  units 
proeured  is  a  ehange  to  the  program  itself,  rather  than  a  ehange  to  the  estimated  eost  of 
the  original  program. 

This  study  ealeulates  eost  and  sehedule  growth  as  a  pereentage  of  the  initial 
estimate,  oeeurring  during  the  development  period  of  the  spaee  system.  The  ealeulations 
inelude  quantity  and  inflation  adjustments,  where  appropriate.  See  Chapters  III  and  IV 
for  further  details  on  ealeulating  eost  and  sehedule  growth. 

IQS.  What  characteristics  of  the  program  or  acquisition  environment  are  good 
predictors  of  cost  and  schedule  growth? 

In  order  to  aeeurately  prediet  eost  and  sehedule  growth,  this  study  needs  to 
identify  eharaeteristies  of  the  program  and  aequisition  environment  that  eould  be 
predietors  of  eost  growth.  One  of  the  primary  goals  of  this  study  is  to  identify  the  best 
predietors  of  growth  and  to  quantitatively  assess  the  relative  impaet  that  these  predietors 
have  on  growth.  The  analysis  examines  numerous  predietor  variables  ineluding: 
eommodity  type,  mission  area,  program  size,  and  prime  eontraetor,  in  order  to  establish  a 
set  of  models  for  predieting  spaee  system  eost  and  sehedule  growth.  See  Chapter  III  for 
more  details  on  the  predietor  variables  analyzed. 

IQ4.  What  are  the  current  methodologies  for  predicting  cost  and  schedule  growth  ? 

Assessing  the  relationship  between  the  potential  predietors  and  eost  and  sehedule 
growth  requires  seleeting  an  appropriate  methodology.  Beeause  this  is  an  exploratory 
analysis,  it  is  useful  to  review  other  methodologies  for  modeling  eost  and  sehedule 
growth  in  order  to  determine  if  this  researeh  ean  apply  these  teehniques  in  whole  or  in 
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part.  Additionally,  understanding  the  range  of  methodologies  available  assists  in 
revealing  the  strengths  and  limitations  of  the  methodologies  ineorporated  into  this  study. 
IQS.  How  can  the  cost  and  schedule  growth  models  be  validated? 

By  implementing  model  diagnosties  for  validating  predietive  models,  this  study  is 
able  to  assess  how  aeeurately  the  models  prediet  future  eost  and  sehedule  growth. 
Validating  the  models  ensures  the  robustness  of  the  models  as  a  predietive  tool,  and  that 
the  models  will  be  useful  to  military  and  eivil  spaee  system  eost  estimating  and 
aequisition  eommunities. 

Methodology 

Due  to  the  differenee  in  available  data  for  DoD  and  NASA  spaee  systems,  this 
study  analyzes  military  and  eivil  systems  separately.  The  analysis  of  DoD  spaee  systems 
applies  linear  regression  to  identify  predietors  for  military  spaee  system  eost  growth. 
Unfortunately,  adequate  data  were  not  available  for  assessing  sehedule  growth  for 
military  spaee  systems. 

In  order  to  analyze  NASA  spaee  systems  eost  and  sehedule  growth,  this  study 
uses  a  two-staged  regression  methodology  developed  by  Sipple  (2002).  The  study  uses 
this  two-stage  approaeh  due  to  the  bimodal  nature  of  the  eost  and  sehedule  growth  data 
(see  Chapter  III  for  further  details  on  the  bimodal  distribution  of  the  data).  The  analysis 
adapts  Sipple’s  two-staged  approaeh  by  first  using  logistie  regression  in  order  to  assess 
the  likelihood  that  a  NASA  spaee  system  will  experienee  high  or  low  growth.  The 
seeond  stage  uses  multiple  regression  analysis  in  order  to  model  the  expeeted  amount  of 
growth.  See  Chapter  III  for  further  details  on  the  methodology. 
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Study  Results 

The  DoD  cost  growth  analysis  reveals  that  communications  missions,  ground 
equipment,  firm- fixed  price  contracts,  and  increased  program  manager  tenure  are  all 
predictive  of  lower  cost  growth. 

The  NASA  cost  growth  analysis  found  that  larger  program  size  decreased  the 
likelihood  of  being  a  high  cost  growth  program,  where  as  more  massive  spacecrafts  and 
microgravity  missions  increased  the  likelihood  of  being  a  high  cost  growth  program.  For 
those  NASA  programs  that  are  likely  to  experience  high  cost  growth,  the  amount  of  cost 
growth  increases  for  those  programs  from  a  space  physics  mission.  For  NASA  programs 
in  which  the  logistic  models  predict  that  low  cost  growth  is  likely,  program  start  date  is 
the  best  predictor  of  quantity  of  cost  growth,  with  more  recent  programs  associated  with 
lower  cost  growth. 

The  NASA  schedule  growth  analysis  found  that  larger  programs  (measured  in 
size  of  budget)  are  more  likely  to  experience  high  schedule  growth.  For  those  programs 
likely  to  experience  high  schedule  growth,  the  linear  regressions  reveal  that  those 
programs  developed  by  JPL  or  an  International  developer  (outside  of  the  U.S.)  experience 
a  greater  quantity  of  schedule  growth.  For  those  programs  likely  to  experience  low 
schedule  growth,  those  developed  by  Northrop  Grumman  are  associated  with  increased 
schedule  growth,  where  as  those  space  systems  developed  by  Johns  Hopkins  are 
associated  with  a  reduced  quantity  of  schedule  growth.  See  Chapters  IV  and  V  for  more 
details  on  the  study’s  results. 
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Organization  of  the  Study 

This  chapter  included  an  overview  of  the  problem  area,  the  research  and 
investigative  questions,  and  the  methodology.  Chapter  II  presents  a  literature  review 
which  begins  to  explore  Investigative  Questions  I  -4  on  defining  space  systems, 
ealeulating  eost  growth,  identifying  potential  predictors  of  cost  growth,  and  identifying 
methodologies  for  prediction  cost  growth.  Chapter  III  provides  a  detailed  discussion  on 
the  data  and  methodology,  concluding  the  diseussion  on  Investigative  Questions  1-4,  as 
well  as  addressing  Investigative  Question  5  on  validation  methods.  Chapter  IV  details 
the  preliminary,  logistic,  and  multiple  regression  analysis  of  the  data,  as  well  as  detailing 
the  diagnostic  tools  used  for  validating  the  models.  Chapter  V  concludes  the  study  with  a 
discussion  of  the  results. 
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II,  Literature  Review 


Chapter  Overview 

This  literature  review  examines  previous  aequisition  cost  and  schedule  growth 
studies  in  order  to  gain  a  greater  understanding  of  space  system  cost  and  schedule  growth 
and  how  best  to  analyze  it.  In  so  doing,  this  chapter  begins  the  study’s  exploration  of 
four  of  the  investigative  questions  introduced  in  Chapter  I; 

1 .  Which  systems  should  be  considered  “space  systems”? 

2.  What  are  the  current  methods  for  calculating  cost  and  schedule  growth? 

3.  What  characteristics  of  the  program  or  acquisition  environment  are  good 
predictors  of  cost  and  schedule  growth? 

4.  What  are  the  current  methodologies  for  predicting  cost  and  schedule  growth? 

This  literature  review  examines  these  four  investigative  questions  by  establishing  the 
scope  of  the  literature,  identifying  definitions  for  “space  system,”  detailing  methods  for 
calculating  cost  and  schedule  growth,  discussing  candidate  predictor  variables,  and 
evaluating  past  methodologies. 

Literature  Scope 

As  discussed  in  Chapter  I,  this  study  focuses  on  cost  and  schedule  growth  for 
military  and  civil  space  systems.  Due  to  the  limited  number  of  space  system  cost  and 
schedule  growth  studies,  this  literature  review  also  considers  research  focusing  on 
acquisition  of  all  types  of  Major  Defense  Acquisition  Programs  (MDAPs).^  Recent 

^  Major  Defense  Acquisition  Programs  (MDAPs)  are  defense  programs  that  have  an  estimated  Research, 
Development,  Test  and  Evaluation  (RDT&E)  cost  of  over  $365  million  or  an  estimated  Procurement  cost 
of  more  than  $2.19  billion.  Additionally,  high  interest  programs  not  meeting  these  requirements  can  be 
designated  as  MDAPs  by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
USD(AT&L)  (“Life,”  2004). 
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studies  analyzing  space  system  cost  growth  are  primarily  qualitative  (Defense  Science, 
2003;  Salas,  2004;  Government,  2006).  Although  the  space  acquisition  community  has 
placed  considerable  effort  in  modeling  cost  (Bearden,  2000/2001;  Tieu  et  ah,  2000),  there 
is  a  dearth  of  quantitative  analysis  on  space  system  cost  growth.  Recent  quantitative 
studies  on  space  system  cost  growth  appear  to  be  limited  to  two  NASA  studies:  Tyson  et 
al.’s  (1992a)  and  Schaffer’s  (2004). 

However,  cost  growth  studies  examining  all  MDAPs  have  been  plentiful.  Most  of 
these  studies  use  Selected  Acquisition  Reports  (SARs)  (which  contain  program  costs  and 
cost  estimates)  as  their  source  for  cost  data.  All  MDAPs  are  required  to  submit  SARs  to 
Congress  annually,  with  “exception  SARs”  submitted  on  a  quarterly  basis  if  major 
changes  occur  (Hough,  1992).  Most  DoD  cost  growth  studies  use  SAR  data  to  examine 
weapon  system  cost  growth  across  multiple  services  and  multiple  platforms.  These 
studies  may  include  cost  growth  data  for  space  systems;  however,  these  studies  do  not 
conduct  separate  analyses  on  the  relationship  between  predictors  of  cost  growth  and 
space  system  cost  growth. 

Cost  growth  literature  for  defense  programs  and  space  systems  far  exceeds  the 
available  literature  for  schedule  growth.  There  are  many  possible  reasons  for  the  limited 
number  of  schedule  growth  studies.  Cashman  (1995)  identifies  attitudes  regarding  lack 
of  control  over  schedule  and  beliefs  that  schedule  growth  on  one  program  does  not 
translate  to  other  programs  as  the  reasons  for  limited  research.  Cross  (2006)  points  out 
other  limitations,  especially  when  using  SAR  data,  including  minimal  reporting 
requirements  and  inconsistencies  in  schedule  baselines.  Quantitative  schedule  growth 
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studies  examine  all  defense  weapons  systems;  there  do  not  appear  to  be  any  focused 
solely  on  space  systems. 

DoD  cost  and  schedule  growth  studies  referenced  in  this  literature  review  include, 
but  are  not  limited  to,  studies  performed  by  the  RAND  Corporation,  Institute  for  Defense 
Analyses  (IDA),  Management  Consulting  and  Research  (MCR),  Air  Force  Institute  of 
Technology  (AFIT)  students.  Naval  Postgraduate  School  (NPS)  students,  and  the  Office 
of  the  Secretary  of  Defense  (OSD)  Cost  Analysis  Improvement  Group  (CAIG).  The  DoD 
cost  and  schedule  growth  studies  conducted  by  RAND  and  IDA  consist  of  a  series  of 
analyses  performed  in  the  early  1990s  (Drezner  et  ah,  1993;  Tyson  et  ah,  1992b,  Drezner 
and  Smith,  1990),  as  well  as  a  follow-on  series  performed  in  the  mid-2000s  (Arena  et  ah, 
2006;  McNicol,  2005).  Both  the  current  and  historical  RAND  and  IDA  cost  and  schedule 
growth  studies  primarily  center  on  descriptive  statistics.^ 

In  addition  to  the  recent  RAND  and  IDA  cost  and  schedule  growth  studies,  a 
series  of  AFIT  theses  analyze  DoD  weapon  system  cost  and  schedule  growth.  Unlike  the 
RAND  and  IDA  studies,  the  AFIT  theses  employ  more  rigorous  statistical  methods,  such 
as  logistic  and  multiple  regression  analysis.  These  statistical  techniques  model  cost  and 
schedule  growth,  thus  enabling  the  researcher  to  make  predictions  regarding  cost  and 
schedule  growth  for  systems  not  included  in  the  sample.  Another  key  difference  is  that 
the  AFIT  theses  examine  subsets  of  total  cost  growth,  for  example,  schedule  or 
engineering  changes  (Foreman,  2007;  Cross,  2006;  Monaco,  2005;  Genest,  2004;  Lucas, 
2004;  McDaniel,  2004;  Rossetti,  2004;  Bielecki,  2003;  Moore,  2003;  Sipple,  2002). 

^  Descriptive  statistics  employ  visual  methods  in  order  to  summarize  the  characteristics  of  the  data  in  the 
sample,  as  opposed  to  inferential  statistics  which  use  the  data  to  make  predictions  about  the  population 
from  which  the  sample  was  drawn  (McClave  et  ah,  2005:5) 
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Defining  “Space  System” 

In  order  to  analyze  spaee  system  eost  and  sehedule  growth,  this  study  must 
eonsider  Investigative  Question  1,  “Whieh  systems  should  be  eonsidered  ‘space 
systems’?”  Tyson  et  ah,  in  their  study  on  NASA  and  DoD  space  systems,  define  a  space 
system  as  “a  collection  of  integrated  components  to  achieve  a  specific  purpose  beyond 
the  Earth’s  atmosphere,  such  as  Earth  observation  or  planetary  exploration”  (1992a:4). 
Erom  this  definition,  those  systems  operating  in  the  space  environment  ought  to  be 
considered  space  systems.  Thus,  satellites  and  non-orbiting  spacecraft  are  clearly  space 
systems.  But  what  about  systems  designed  to  support  these  missions,  such  as  launch 
vehicles  or  satellite  terminals?  Are  they  not  also  “space  systems”? 

Sellers  et  al.  (2004),  in  Understanding  Spaced  describe  the  space  mission 
architecture  as  being  composed  of  six  parts:  the  mission,  the  spacecraft,  the  trajectories 
and  orbits,  the  launch  vehicle,  the  mission  operations  systems,  and  the  mission 
management  and  operations.  This  interpretation  is  considerably  broader  than  Tyson  et 
al.'s  (1992a)  by  including  the  associated  launch  vehicles  and  infrastructure. 

Another  approach  to  defining  “space  system”  is  to  examine  which  systems  are 
operated  and  acquired  by  the  space  segments  of  the  DoD.  Air  Eorce  Space  Command 
(AESPC)  operates  all  space  forces  for  the  Air  Eorce.  AESPC  operations  involve  the 
control  of  space-based  satellites,  ground-based  terminals  and  early  warning  radars,  and 
strategic  missiles.  The  Space  and  Missile  Systems  Center  (SMC)  is  the  acquisition  arm 
for  AESPC  and  is  the  primary  acquirer  for  DoD  space  systems  (“Air,”  2007).  SMC 

’  The  Understanding  Space  textbook  is  widely  used  in  Air  Force  space  education,  including  courses  taught 
at  the  Air  Force  Academy,  the  Air  Force  Institute  of  Technology,  and  the  Space  and  Missile  Systems 
Center. 
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acquires  satellite  systems,  launeh  vehicles,  land-based  ballistie  missiles,  and  spaee  system 
ground  equipment  (History,  2003).  Thus,  from  the  eurrent  DoD  perspective,  spaee 
system  operations  and  aequisition  not  only  includes  all  of  the  systems  provided  in  Sellers 
et  al.’s  (2004)  interpretation,  but  also  ineludes  strategie  missiles.  Similar  to  the  Air 
Foree,  the  Army  consolidates  its  missile  aequisition  and  operations  with  its  other  space 
system  aequisition  and  operations  under  a  single  organization,  the  U.S.  Army  Spaee  and 
Missile  Defense  Command  (“U.S.,”  2007). 

How  have  other  cost  and  schedule  growth  studies  defined  “spaee  systems”? 
Unfortunately,  most  DoD  studies  avoid  defining  “spaee  system”  when  referring  to 
weapon  system  types,  choosing  instead  to  segment  these  types  of  systems  into  separate 
eategories  sueh  as  “satellites,”  “launeh  vehieles,”  and  “missiles”  (Arena  et  ah,  2006; 
Tyson  et  ah,  1992b;  Wolf,  1990).  As  for  the  limited  number  of  DoD  studies  that 
referenee  spaee  systems,  satellites  and  launeh  vehieles  are  eonsistently  treated  as  space 
systems,  while  strategie  missile  systems  vary.  MeCrillis  (2003)  separates  strategic 
missiles  from  taetieal  missiles,  ehoosing  to  inelude  strategie  missile  systems  in  the  spaee 
eategory.  Drezner  et  al.  (1993)  eombine  strategic  missiles  with  taetieal  missiles  into  a 
single  “missile”  eategory. 

Cost  and  Schedule  Growth  Definition  and  Calculation 

In  addition  to  exploring  the  possible  definitions  for  “spaee  system,”  this  study 
must  also  Investigative  Question  2,  “What  are  the  eurrent  methods  for  calculating  cost 
and  schedule  growth?”  This  seetion  first  examines  eost  growth,  whieh  includes  three 
main  elements:  the  cost  growth  formula,  the  varianee  types,  and  the  inflation  and  quantity 
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adjustments.  This  section  concludes  with  an  examination  of  schedule  growth 
calculations. 

Cost  Growth  Formula 

Cost  growth  is  a  comparison  of  cost  variance  to  the  original  cost  estimate.  Cost 
variance  is  defined  as  the  difference  between  planned  cost  (original  baseline  cost 
estimate)  and  actual  cost  (or  updated  cost  estimate)  (Department,  1980).  Cost  growth 
studies  calculate  cost  growth  in  one  of  two  ways.  The  first  approach  is  to  calculate  cost 
growth  as  a  percentage  of  the  original  cost  estimate  (McNichols  and  McKinney,  1981; 
Pannell,  1994;  Bielecki  and  White,  2005): 

Cost  Growth  =  ( Actual-Estimate)  (1) 

Estimate 

Equation  1  provides  cost  growth  as  a  percentage,  where  a  value  of  “zero”  means  the  is  no 
cost  growth,  a  negative  value  means  that  the  actual  cost  is  less  than  the  planned  costs,  and 
a  positive  value  means  that  the  actual  cost  is  greater  than  the  planned  cost. 

The  second  approach  is  to  calculate  cost  growth  as  a  cost  growth  factor  (CGE)  or 
cost  growth  ratio  (CGR).  This  approach  simply  divides  the  actual  cost  (or  updated  cost 
estimate)  into  the  planned  cost  (Arena  et  ah,  2006;  McCrillis,  2003;  Tyson  et  ah,  1994; 
Drezner  et  ah,  1993): 


Cost  Growth  =  (Actual) 
Estimate 


(2) 


23 


Equation  2  provides  cost  growth  as  a  factor,  where  a  value  equal  to  one  means  there  is  no 
cost  growth,  a  value  less  than  one  means  that  the  actual  cost  is  less  than  the  planned  cost, 
and  a  value  greater  than  one  means  that  the  actual  cost  is  greater  than  the  planned  cost. 

In  addition  to  identifying  approaches  to  calculate  cost  growth,  one  must  also 
define  the  components  of  the  cost  growth  formula:  estimated  cost  and  actual  cost.  DoD 
cost  growth  studies  typically  use  the  Development  Estimate  (DE),  which  is  the  estimate 
submitted  for  Milestone  B,  as  the  baseline  estimate.*  The  DE  is  the  best  estimate  to 
capture  the  program’s  planned  cost  because  at  this  point  the  major  design  and  capability 
trade-offs  have  occurred  and  the  program  office  is  ready  to  begin  system  development 
(“Defense  Acquisition,”  2007;  Jarvaise  et  ah,  1996;  Department,  1980). 

While  cost  growth  studies  agree  on  which  estimate  to  use  as  the  baseline  cost, 
there  is  some  variation  in  which  estimate  to  use  as  the  actual  cost.  A  number  of  studies 
use  the  current  estimate  as  actual  cost  (McCrillis,  2003;  Drezner  et  al.,  1993;  McNichols 
and  McKinney,  1981).  By  using  the  current  estimate,  not  all  programs  will  be  at  the 
same  stage  in  their  development;  some  programs  will  be  in  the  beginning  of  their 
development,  some  programs  will  be  near  the  end  of  their  development,  and  some  may 
even  be  complete.  Those  programs  that  are  near  the  end  of  development  or  are  complete 
will  have  incurred  a  higher  proportion  of  their  cost  growth  than  those  at  the  beginning. 
Basing  calculations  on  programs  at  different  stages  of  development  may  cause  those  at 
the  beginning  of  their  development  process  to  bias  the  results  because  they  have  not  yet 
experienced  most  of  their  cost  growth.  Arena  et  al.  (2006)  show  that  programs  continue 

*  Milestone  B,  previously  known  as  Milestone  II,  marks  program  initiation  and  is  the  point  where  a  DoD 
weapon  system  enters  System  Development  and  Demonstration. 
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to  incur  cost  growth  until  approximately  70-80%  of  their  development  and  produetion  has 
been  completed.  Because  of  this  tendency  to  incur  cost  growth  through  the  later  stages  of 
development  and  produetion,  Arena  et  al.  (2006)  ehoose  to  include  only  eompleted 
programs;  therefore,  their  analysis  consists  of  only  final  system  costs. 

Most  DoD  cost  growth  studies,  however,  use  a  mix  of  completed  and  on-going 
programs.  McNieol  (2005)  sets  a  minimum  requirement  for  programs  to  be  at  least  three 
years  past  Milestone  B  to  qualify  for  inclusion  in  his  cost  growth  study.  Other 
researchers  place  a  limit  on  the  time  frame  by  using  costs  from  Milestone  B  up  to  the 
initial  operational  capability  (IOC)  date,  but  do  not  inelude  costs  occurring  after  IOC 
(Tyson  et  ah,  1992b;  Wolf,  1990). 

Variance  Types 

As  previously  mentioned,  most  DoD  cost  growth  studies  use  SARs  as  their  souree 
for  cost  data.  SARs  include  the  original  DE  and  current  estimates  (CE).  Differenees 
between  the  DE  and  the  CE  are  ealled  “varianees”  and  are  separated  into  seven 
categories:  Economic,  Quantity,  Schedule,  Engineering,  Estimating,  Support,  and  Other 
(Hough,  1992;  Department,  1980).  See  Chapter  III  for  a  more  detailed  discussion  of  the 
SAR. 

Most  studies  agree  that  variances  due  to  inflation  (Economic  category)  or  quantity 
constitute  unforeseen  eost  growth,  and  thus  ehoose  to  adjust  either  the  DE  or  the  CE  for 
these  types  of  cost  growth  (see  section  on  Inflation  and  Quantity  Adjustments).  RAND, 
IDA,  AEIT,  and  NFS  studies  typically  combine  the  other  five  cost  variances 
(Engineering,  Sehedule,  Support,  Estimating,  and  Other),  focusing  on  total  cost  growth 
adjusted  for  quantity  and  inflation  (Arena  et  ah,  2006;  Drezner  et  ah,  1993;  Tyson  et  ah. 
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1994;  Tyson  et  al,  1992b;  Moore,  2003;  Genest,  2004;  Lucas,  2004;  Wolf,  1990).  Fast 
(2007),  in  “Sources  of  Program  Cost  Growth,”  argues  that  because  Quantity,  Economic, 
Estimating,  and  Other  variances  represent  changes  that  are  beyond  the  cost  estimator’s 
ability  to  forecast,  these  categories  do  not  constitute  actual  cost  growth;  rather,  it  is  the 
combination  of  Engineering,  Schedule,  and  Support  that  constitutes  actual  cost  growth. 
DoD  cost  growth  studies  examining  total  cost  growth  have  yet  to  adopt  this 
recommendation  to  exclude  the  Estimating  and  Other  categories  and  focus  solely  on  the 
combination  of  Engineering,  Schedule,  and  Support. 

Instead  of  considering  total  cost  growth,  several  AEIT  theses  examine  these 
categories  separately  in  order  to  isolate  the  predictors  for  these  individual  aspects  of  cost 
growth.  Sipple  (2002)  examines  cost  growth  due  to  Engineering  varinaces  within  the 
Research,  Development,  Test,  and  Evaluation  (RDT&E)  appropriation.  Bielecki  (2003) 
builds  on  this  work  by  individually  examining  the  RDT&E  appropriation  for  the  four 
other  categories:  Estimating,  Schedule,  Support,  and  Other.  Rossetti  (2004)  complements 
both  sets  of  work  by  examining  the  Estimating  and  Support  categories  for  the 
Procurement  appropriation,  whereas  McDaniel  (2004)  analyzes  the  Engineering  and 
Schedule  categories  for  the  Procurement  appropriation. 

Although  using  SAR  variance  categories  is  the  primary  method  for  analyzing  cost 
growth,  several  studies  use  the  variance  categories  created  by  the  DoD  Office  of  the 
Director  for  Program  Analysis  and  Evaluation  (PA&E).  PA&E  uses  SAR  data  to  create 
its  own  cost  database,  dividing  variances  between  those  that  are  attributed  to  mistakes 
and  those  that  are  attributed  to  decisions.  The  mistake  variance  is  further  divided  into 
five  subcategories:  production,  development  and  engineering,  logistics  support,  schedule 
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and  management  faetors,  and  other.  Similarly,  the  deeision  varianee  is  divided  into  five 
subeategories:  requirements,  sehedule,  logisties  support,  external  faetors,  and  other.  IDA, 
CAIG,  and  NFS  have  taken  advantage  of  this  database  in  order  to  eharaeterize  cost 
variances  attributable  to  decisions  and  mistakes  (McNicol,  2005;  McCrillis,  2003; 

Pannell,  1994). 

Inflation  and  Quantity  Adjustments 

DoD  cost  growth  studies  use  one  of  two  methods  to  adjust  for  inflation:  1)  convert 
all  program  costs  to  base-year  dollars  for  that  system  or  2)  adjust  costs  for  all  programs  to 
a  standard  base  year.  Most  studies,  including  those  done  by  RAND,  IDA,  and  MCR,  use 
base-year  dollars  to  adjust  for  inflation  when  calculating  cost  growth  (Arena  et  ah,  2006; 
Drezner  et  ah,  1993;  Tyson  et  ah,  1992b;  McNichols  and  McKinney,  1981).  Recent 
studies  using  the  PA&E  database  adjust  for  inflation  by  converting  all  program  cost  data 
to  Fiscal  Year  (FY)  2000  constant  dollars  (McNicol,  2005;  McCrillis,  2003).  AFIT 
theses  take  a  similar  approach,  converting  base-year  dollars  for  each  program  into  a 
standard  base  year  (Fucas,  2004;  Bielecki,  2003;  Sipple,  2002). 

In  addition  to  adjusting  for  inflation,  DoD  cost  growth  studies  also  adjust  for 
variances  due  to  quantity  changes  because  cost  estimators  create  the  DF  with  the  original 
planned  quantity  in  mind  and  do  not  incorporate  adjustments  for  quantity  changes.  To 
adjust  for  quantity,  DoD  cost  growth  studies  take  one  of  two  approaches:  1)  adjust  the  CE 
to  reflect  baseline  quantities  or  2)  adjust  the  DE  to  reflect  current/final  quantities. 
Adjusting  the  CE  to  reflect  baseline  quantities  provides  cost  growth  in  terms  of  the  initial 
cost  estimate  and  prevents  a  “floating  baseline,”  where  the  quantity  used  for  calculations 
changes  from  year  to  year  (Drezner  et  ah,  1993).  On  the  other  hand,  adjusting  the  DE  to 
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reflect  current  quantities  allows  the  researcher  to  modify  estimates  while  keeping  actual 
costs  intact.  When  calculating  total  cost  growth  with  both  RDT&E  and  Procurement 
appropriations,  adjusting  the  DE  to  the  current  quantity  maintains  the  proportion  of 
procurement  cost  to  total  cost  (Arena  et  ah,  2006). 

After  selecting  whether  to  use  the  baseline  or  current  quantity,  the  researcher  must 
then  make  the  necessary  adjustments  to  the  CE  or  DE.  Hough  (1992)  offers  three 
methods  for  performing  this  quantity  normalization; 

1 .  Normalize  using  variance  listed  in  the  SAR  Quantity  category  only, 

2.  Normalize  using  cost-quantity  curves,  thus  adjusting  all  variances  that  occur 
at  other  than  baseline  quantities,  or 

3.  Normalize  using  a  hybrid  approach  by  adjusting  for  quantity-related  variances 
(both  those  listed  in  SAR  Quantity  category  as  well  as  those  listed  in  other 
categories  but  described  as  quantity-related  in  the  narrative  portion  of  the 
SAR)  and  then  adjusting  the  remaining  variance  using  cost-quantity  curves. 

Although  Hough  (1992)  recommends  using  either  cost-quantity  curves  or  the  hybrid 
approach,  most  AEIT  theses  implement  the  first  approach  and  exclude  cost  variances 
listed  in  the  Quantity  category  (Genest,  2004;  Bielecki,  2003;  Sipple,  2002).  One 
exception  is  Abate ’s  (2004)  thesis  on  missile  system  cost  growth  which  uses  the  hybrid 
approach.  DoD  cost  growth  studies  performed  by  RAND,  IDA,  and  CAIG  implement 
cost-quantity  curves  (also  known  as  learning  curves,  cost  improvement  curves,  or  price 
improvement  curves)  either  directly  or  through  the  hybrid  approach  (Arena  et  ah,  2006; 
McNicol,  2005;  McCrillis,  2003;  Drezner  et  ah,  1993;  Tyson  et  ah,  1992b). 
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Calculating  Schedule  Growth 

As  with  cost  growth,  schedule  growth  is  most  eommonly  ealculated  either  as  a 
pereentage  of  the  planned  length  or  as  a  ratio  between  actual  and  planned  length 
(Foreman,  2007;  Cross,  2006;  Monaco,  2005;  Wolf,  1990).  Other  methods  for 
ealculating  schedule  growth  inelude  measuring  just  the  raw  inerease  in  length  in  terms  of 
how  mueh  the  actual  length  exeeeded  the  planned  length  (Drezner  and  Smith,  1990).  In 
addition  to  measuring  schedule  growth  in  terms  of  length,  Cashman  (1995)  also  provides 
ealeulations  for  schedule  growth  in  dollar  terms  and  in  frequeney  of  schedule  changes. 
Predictor  Variables 

In  addition  to  presenting  methods  for  ealeulating  the  response  variable  (cost  or 
schedule  growth),  the  literature  reveals  an  array  of  possible  predictor  variables  to  assist  in 
answering  Investigative  Question  3,  “What  characteristics  of  the  program  or  acquisition 
environment  are  good  predictors  of  cost  and  schedule  growth?”  This  discussion  begins 
by  reviewing  quantitative  DoD  cost  growth  studies  for  predictors  of  cost  and  schedule 
growth,  which  are  grouped  into  three  categories:  program  attributes,  management 
practices  and  aequisition  strategies,  and  external  factors.  The  diseussion  concludes  with 
an  examination  of  predictors  identified  by  qualitative  studies  foeused  exclusively  on 
space  systems. 

Program  Attributes 

Cost  and  schedule  growth  studies  compare  a  variety  of  program  attributes,  in 
order  to  identify  those  program  attributes  that  consistently  eorrespond  with  high  or  low 
growth.  The  primary  program  attributes  assoeiated  with  cost  growth  are  eommodity  type 
and  program  size,  whereas  the  primary  characteristie  assoeiated  with  sehedule  growth  is 
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program  volatility.  Both  cost  growth  and  schedule  growth  studies  identify  degree  of 
teohnieal  diffieulty  as  a  predietor  of  growth. 

Commodity  Type.  A  number  of  studies  eompare  eost  growth  aeross  eommodity 
types,  sueh  as  aireraft,  ships,  land  vehicles,  and  missiles.  As  diseussed  herein,  there  is 
some  differentiation  on  the  eommodity  elassifieation;  for  example,  some  researehers 
choose  to  plaee  satellites,  launch  vehicles,  and  strategie  missiles  into  separate  commodity 
classes  (Arena  et  al.,  2006)  while  others  choose  to  eombine  these  into  a  single  space 
eommodity  elass  (MeCrillis,  2003).  Studies  that  eonsider  spaee  as  a  single  eommodity 
class  consistently  find  space  systems  to  be  assoeiated  with  higher  eost  growth  (MeDaniel, 
2004:83-97;  Rossetti,  2004:93-99;  MeCrillis,  2003).  As  seen  in  Figure  6,  spaee  systems 
experience  eonsiderably  greater  total  eost  growth  than  other  types  of  DoD  programs. 


Aircraft  C4I(24)  Missile  (34)  Ship  (19)  Space  (9)  Ground 
(42)  Combat 

Commodity  Type  (^4) 

Note:  Numbers  in  parentheses  are  numbers  of  programs. 

Figure  6.  Total  Program  Cost  Growth  by  Commodity  (MeCrillis,  2003) 
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Program  Size.  When  considering  program  size,  cost  growth  studies  consistently 
find  that  smaller  programs  (that  is,  lower  cost  programs)  have  higher  cost  growth  than 
larger  programs  (McCrillis,  2003;  Dameron  et  ah,  2002;  Pannell,  1994;  Drezner  et  ah, 
1993).  As  seen  in  Figure  7,  programs  with  Milestone  II  estimates  (DEs)  greater  than  $10 
billion  are  unlikely  to  experience  more  than  50%  cost  growth,  whereas  a  number  of 
systems  with  estimates  below  $10  billion  have  experienced  greater  than  50%  cost  growth. 
Drezner  et  al.  (1993)  offer  three  possible  explanations  for  smaller  programs  incurring 
high  cost  growth:  1)  oversight  is  often  less  for  smaller  programs,  2)  equivalent  increases 
in  cost  are  proportionally  greater  for  smaller  programs,  and  3)  R&D  costs  (which  tend  to 
have  higher  cost  growth  than  procurement  costs)  consist  of  a  greater  proportion  of  the 
total  cost  for  smaller  programs.  While  Tyson  et  al.  (1992a)  find  this  relationship  to  be  the 
case  for  DoD  space  programs,  they  observe  that  NASA  space  programs  are  the  opposite, 
with  larger  NASA  programs  experiencing  higher  cost  growth.  However,  a  more  recent 
study  of  NASA  programs  contradicts  this  conclusion,  finding  that  both  NASA  and  DoD 
programs  experience  lower  cost  growth  as  program  size  increases  (Schaffer,  2004). 
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Figure  7.  Total  Cost  Growth  by  Program  Size  (MeCrillis,  2003) 


Program  Volatility.  DoD  schedule  growth  studies  find  that  volatility  is  one  of  the 
main  factors  associated  with  higher  schedule  growth.  These  studies  measure  volatility  in 
a  variety  of  ways  including  the  number  of  changes  to  the  original  estimate  (Foreman, 
2007;  Cross,  2006),  funding  instability  (Foreman,  2007;  Drezner  and  Smith,  1990), 
technical  problems,  and  design  changes  (Cashman,  1995;  Drezner  and  Smith,  1990). 

Technical  Difficulty.  DoD  cost  and  schedule  growth  studies  evaluate  the  degree 
of  technical  difficulty  by  comparing  cost  or  schedule  growth  of  new  programs  to  those 
that  have  predecessor  programs.  One  would  expect  that  new  programs  would  be  more 
technically  challenging  (and  thus  have  higher  cost  or  schedule  growth)  than  modification 
programs  that  build  upon  predecessor  programs.  Drezner  et  al.  (1993)  find  that 
modification  programs  experience  lower  average  total  cost  and  lower  cost  growth,  as 
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shown  in  Table  1.  Other  studies  also  eonsistently  show  this  scenario  to  be  the  case,  with 
new  programs  having  higher  cost  or  schedule  growth  than  modification  programs  (Cross, 
2006;  McNicol,  2005;  Monaco,  2005;  Tyson  et  al,  1992b). 


Table  1.  Modifications  Versus  New  Programs  (Drezner  et  ah,  1993) 


Cost 

Growth 

Factor 

Number  of 
Observations 

Average  Program 
Cost  (billions, 
FY90$) 

Average  Age 
(years  past 
EMD) 

Modification 

1.16 

36 

4.0 

8.9 

New  Start 

1.21 

84 

6.1 

9.7 

Management  Practices  and  Acquisition  Strategies 

DoD  cost  and  schedule  growth  studies  also  examine  the  impact  of  management 
practices  and  acquisition  strategies  on  growth.  These  practices  and  strategies  include 
acquiring  service,  schedule  characteristics,  contract  type,  and  prototyping. 

Acquiring  Service.  DoD  cost  growth  studies  disagree  on  the  impact  of  service 
type  on  cost  growth.  In  their  study  for  MCR,  McNichols  and  McKinney  (1981)  find  that 
the  Army  tends  to  have  higher  cost  growth  than  the  other  services.  This  finding  is 
confirmed  by  a  RAND  study  in  1993  (Drezner  et  al.,  1993)  and  more  recently  by  an  IDA 
study  in  2005  (McNicol,  2005).  Recent  AFIT  theses,  on  the  other  hand,  are  mixed  with 
some  finding  Navy  involvement  to  be  associated  with  higher  cost  growth  (McDaniel, 
2004),  while  others  find  Navy  involvement  to  be  associated  with  lower  cost  growth 
(Rossetti,  2004;  Bielecki,  2003).  To  further  confound  these  results,  a  number  of  studies 
have  found  that  there  is  no  difference  in  cost  growth  among  services  (Arena  et  ah,  2006; 
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Pannell,  1994).  Pannell  (1994)  explains  that  studies  relying  on  data  from  the  early  1980s 
are  likely  to  find  Army  programs  experieneing  high  cost  growth  because  the  Army  was 
going  though  its  modernization  program.  He  suggests  that  more  recent  data  reflect  the 
completion  of  Army  modernization  and  that  the  Army  is  now  better  able  to  control 
program  costs.  Drezner  et  al.  (1993)  explain  that  Army  programs  are  more  likely  to  have 
higher  cost  growth  because  the  Army  programs  in  their  dataset  tend  to  be  smaller  and 
older  than  the  Navy  and  Air  Force  programs.  Thus,  some  of  the  cost  growth  attributed  to 
service  may  actually  be  due  to  size  and  age. 

Schedule  Characteristics.  AFIT  theses  find  that  schedule  characteristics,  such  as 
the  length  of  the  Research  and  Development  (R&D)  Phase  or  the  length  of  the  Production 
Phase,  are  good  indicators  of  both  the  likelihood  of  cost  growth  and  the  amount  of  cost 
growth.  The  most  consistent  finding  is  a  positive  relationship  between  the  length  of  the 
R&D  phase  and  cost  growth;  that  is,  longer  R&D  phases  correspond  with  increased  cost 
growth  (Lucas,  2004;  McDaniel,  2004;  Bielecki,  2003;  Sipple,  2002).  AFIT  studies  find 
lower  schedule  growth  for  programs  with  a  Phase  A,  programs  that  have  a  longer  Phase 
A  plus  planned  Phase  B,  and  for  programs  that  have  their  Milestone  C  prior  to  their 
Initial  Operational  Capability  (IOC)  date  (Foreman,  2007;  Cross,  2006;  Monaco,  2005). 

Contract  Type.  As  for  acquisition  strategies,  IDA  finds  that  multi-year 
procurement  contracts  and  development  contracts  that  include  incentives  tend  to  have 
lower  cost  growth  (Tyson  et  ah,  1992b).  McNicol  (2005)  finds  that  total  package 
procurement  contracts  result  in  higher  cost  growth,  and  Rossetti  (2004)  finds  that  fixed- 
price  contracts  decrease  the  likelihood  of  cost  growth  occurring. 
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Prototyping.  The  literature  is  mixed  on  the  impaet  of  prototyping  on  cost  growth. 
IDA  finds  prototyping  to  be  an  effective  tool  for  reducing  cost  growth  (Tyson  et  al., 
1992b);  however,  RAND  finds  that  programs  with  prototyping  experience  higher  cost 
growth  (Drezner  et  al.,  1993). 

External  Factors 

External  factors  impacting  cost  and  schedule  growth  include  acquisition  reform, 
political  party,  and  external  guidance.  In  Abate’s  (2004)  study  on  the  impact  of 
acquisition  reform  on  cost  growth  of  tactical  missiles,  he  finds  that  missile  systems 
reporting  their  Milestone  B  estimate  during  the  post- acquisition  reform  period  (1997- 
2001)  experience  higher  cost  growth  than  those  reporting  in  the  pre-acquisition  reform 
period  (1991-1996).  In  Wolfs  (1990)  study  on  political  impacts  on  cost  and  schedule 
growth,  he  finds  that  both  cost  and  schedule  growth  are  higher  for  programs  that  are 
initiated  during  times  when  the  Democratic  Party  has  a  strong  majority  in  Congress. 
Gounatidis  (2006)  finds  that  a  Democratic  President  correlates  with  reduced  cost 
overruns  for  that  year.^  In  addition,  Gounatidis  (2006)  finds  that  having  the  same 
political  party  control  both  houses  of  Congress  or  having  the  same  political  party  control 
the  Senate  and  Presidency  correlates  with  increased  cost  overruns  for  that  year.  In  their 
study  on  schedule  growth,  Drezner  and  Smith  (1990)  find  that  external  guidance  such  as 
oversight  reviews,  legislation,  and  directives  are  associated  with  higher  schedule  growth. 


^  Instead  of  evaluating  SAR  data  for  total  program  eost  growth,  Gounatidis  examines  annual  eost  growth 
reported  in  Cost  Performanee  Reports  (2006:43). 
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Predictors  from  Space  Studies 


In  addition  to  the  quantitative  DoD  eost  and  sehedule  growth  studies,  several 
ageneies  have  conducted  qualitative  analyses  on  cost  growth  among  space  systems. 

These  qualitative  studies  agree  that  the  main  contributors  to  space  system  cost  growth 
are:  the  increase  in  system  requirements  (requirements  creep),  the  large  number  of  Key 
Performance  Parameters  (KPPs),  the  short  tenure  of  program  managers  (PMs),  the  lack  of 
systems  engineering  expertise,  the  use  of  compressed  schedules,  the  incorporation  of 
immature  technologies,  and  the  use  of  Total  System  Performance  Responsibility  (TSPR) 
contracts  (Defense  Science,  2003;  Government,  2006;  Salas,  2004). 

From  these  findings  on  predictors  of  cost  growth,  a  disparity  has  emerged;  while 
several  of  the  factors  identified  by  these  qualitative  studies  (such  as  contract  type  and 
schedule  characteristics)  appear  in  the  quantitative  studies,  most  of  the  factors  do  not. 
What  could  be  the  cause  of  this  disconnect?  Many  of  the  quantitative  studies  take 
advantage  of  the  data  collected  in  the  SAR,  which  limits  their  pool  of  predictor  variables 
to  those  provided  in  the  SAR.  Additionally,  the  predictors  identified  in  the  qualitative 
studies  may  be  difficult  to  operationalize  (that  is,  difficult  to  measure),  such  as  systems 
engineering  expertise.  However,  many  of  the  factors  operationalize  rather  easily,  such  as 
the  number  of  KPPs,  the  number  (or  growth)  in  requirements,  the  average  tenure  for 
PMs,  and  the  maturity  of  technology.  In  these  cases,  data  availability  is  most  likely  the 

reason  that  quantitative  studies  fail  to  incorporate  these  factors  into  their  analyses. 


***  The  DoD  uses  a  standard  system  to  rate  teehnology  maturity,  known  as  Teehnology  Readiness  Levels 
(TRLs)  (Department,  2005). 
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Methodologies 

The  literature  review  now  turns  to  Investigative  Question  4,  “What  are  the  current 
methodologies  for  predicting  cost  and  schedule  growth?”  The  investigation  provides 
valuable  insight  into  understanding  the  variety  of  techniques  employed  by  cost  and 
schedule  growth  studies,  which  include  case  studies,  graphical  analyses,  and  regression 
analyses. 

Case  Studies 

In  order  to  assess  the  state  of  acquisition  for  national  security  space  programs,  the 
Defense  Science  Board  Task  Force,  led  by  Thomas  Young,  employs  a  qualitative 
approach.  The  Young  Task  Force  mainly  relies  upon  interviews  with  government 
personnel,  retired  government  personnel,  and  contractors.  Its  interviews  span  a  broad 
range  of  the  acquisition  spectrum:  high-level  decision  makers  from  the  Pentagon  and 
AFSPC  responsible  for  strategic  planning;  acquirers  from  SMC  responsible  for 
implementing  daily  acquisition  duties;  and  contractors  from  Boeing  Company,  Lockheed 
Martin,  and  TRW  responsible  for  designing  and  building  space  systems  (Defense 
Science,  2003).  The  Task  Force  augments  its  study  with  a  more  detailed  examination  of 
three  high  profile  space  systems:  Space-Based  Infrared  System  (SBIRS)  High,  Future 
Imagery  Architecture  (FIA),  and  Evolved  Expendable  Eaunch  Vehicle  (EEEV)  (Defense 
Science,  2003). 

In  its  2006  report  on  space  system  cost  estimates,  the  Government  Accountability 
Office  (GAO)  utilizes  a  case  study  methodology  involving  detailed  interviews  with 
program  office  and  contractor  personnel  as  well  as  the  examination  of  documentation  on 
program  cost  and  other  program  aspects.  In  this  study,  GAO  focuses  on  six  programs: 
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Advanced  Extremely  High  Frequeney  (AEHF)  satellite,  EEEV,  Global  Positioning 
System  (GPS)  HE,  National  Polar-orbiting  Operational  Environmental  Satellite  System 
(NPOESS),  SBIRS  High,  and  Wideband  Gapfdler  Satellite  (Government,  2006). 

One  of  the  main  benefits  of  a  case  study  is  the  ability  to  gain  a  depth  of 
understanding  of  the  particular  program(s)  being  examined.  However,  case  studies  are 
time  consuming,  costly,  and  limited  by  small  sample  size.  Additionally,  they  do  not 
provide  a  quantitative  measure  of  the  relative  contribution  the  predictors  make  to  cost 
growth.  For  example,  both  the  Young  Task  Force  (Defense  Science,  2003)  and  the  GAO 
(Government,  2006)  studies  identify  requirements  creep  as  a  contributor  to  cost  growth; 
however,  because  they  are  qualitative  studies,  they  do  not  reveal  how  mueh  cost  growth 
can  be  attributed  to  requirements  ereep.  A  quantitative  study  using  regression  analysis, 
on  the  other  hand,  would  be  able  to  forecast  the  pereent  inerease  in  cost  growth  for  each 
additional  requirement  (or  for  a  given  percent  growth  in  requirements). 

Graphical  Analyses 

Cost  and  sehedule  growth  studies  employ  an  assortment  of  methods  for  displaying 
data  in  graphical  and  tabular  form.  Although  these  methods  do  not  provide  the  reader 
with  a  quantitative  measure  of  the  relationship  between  the  factor  of  interest  and  growth, 
they  are  an  effeetive  means  of  visually  displaying  data  and  identifying  patterns.  One  of 
the  most  eommon  methods  is  the  use  of  bar  graphs  and  histograms,  as  shown  by 
MeCrillis  (2003)  in  Figure  8.  In  this  graph,  McCrillis  divides  total  cost  growth  into 
increments  of  ten  pereent,  and  then  displays  the  number  of  systems  (frequency)  that  falls 
into  each  increment.  From  this  graph,  the  reader  can  gather  that  most  systems  experienee 
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relatively  little  eost  growth  (between  -10%  and  20%).  Interestingly,  a  high  number  of 
systems  experience  extreme  cost  growth  (greater  than  70%). 


-10%  0%  10%  20%  30%  40%  50%  60%  70% 


Cost  Growth  Percent  Ranges 


Figure  8.  Total  Cost  Growth  Distribution  (McCrillis,  2003) 


Presenting  comparisons  in  tabular  form  is  another  common  method  for  conveying 
the  relationship  between  a  single  factor  and  growth.  In  the  case  of  Table  2,  cost  growth 
factors  are  displayed  by  weapon  system  type  (commodity  class).  Table  2  allows  for 
quick  comparisons;  for  example,  electronics  experience  relatively  low  cost  growth  (CGF 
1.23),  where  as  missiles  experience  relatively  high  cost  growth  (CGF  1.52).  Although 
displaying  data  in  graphical  and  tabular  form  allows  for  quick  identification  of  patterns 
and  trends,  this  methodology  limits  the  researcher’s  ability  to  interpret  the  data.  Graphs 
and  tables  provide  summaries  of  the  data  in  the  sample;  however,  they  neither  reveal 
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whether  the  relationships  in  the  data  are  statistieally  significant  nor  do  they  provide  the 
researcher  with  the  ability  to  make  predictions  about  the  relationships  in  the  population 
from  which  the  data  are  drawn. 


Table  2.  Cost  Growth  Factor  by  Commodity  Class  (Arena  et  al.,  2006) 


Commodity 

Mean 

Standard 

Deviation 

Number  of 
Observations 

Aircraft 

1.35 

0.24 

9 

Cruise  missiles 

1.64 

0.40 

4 

Electronic  aircraft 

1.52 

0.47 

5 

Electronics 

1.23 

0.33 

12 

Helicopters 

1.76 

0.21 

3 

Eaunch  vehicles 

2.30 

N/A 

1 

Missiles 

1.52 

0.38 

8 

Other 

1.40 

N/A 

1 

Satellites 

1.55 

0.57 

2 

Vehicles 

1.67 

N/A 

1 

Regression  Analyses 

The  preferred  quantitative  methodology  among  DoD  cost  and  schedule  growth 
studies  is  regression  analysis  (Foreman,  2007;  Gounatidis,  2006;  McNicol,  2005;  Tyson 
et  ah,  1994;  Tyson  et  ah,  1992a;  Wolf,  1990).  DoD  cost  and  schedule  growth  studies 
employ  three  types  of  regression  analyses:  simple  linear  regression,  multiple  regression, 
and  logistic  regression.  Simple  linear  regression  quantitatively  describes  a  linear 
relationship  (Figure  9)  between  two  variables:  a  single  predictor  variable  (for  example, 
size  of  program)  and  the  response  variable  (cost  growth).  It  does  so  with  a  straight-line 
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equation  that  expresses  the  response  variable  (y)  as  a  funetion  of  the  predietor  variable  (x) 
(Sehwab,  2005). 


Figure  9.  Linear  Relationship 

However,  simple  linear  regression  limits  the  researeher  to  examining  the  effeets 
of  one  variable  at  a  time.  Beeause  there  are  numerous  possible  predietors  of  eost  or 
sehedule  growth,  multiple  regression  is  a  more  effeetive  tool.  Multiple  regression 
eaptures  the  relationship  between  multiple  predictor  variables  and  a  response  variable. 
With  multiple  regression,  one  can  determine  the  relationship  between  a  predictor  variable 
and  response  variable  while  controlling  for  the  effects  of  other  predictor  variables 
(Schwab,  2005).  For  example,  to  examine  the  relationship  between  cost  growth  and 
acquiring  service,  one  might  create  a  simple  linear  regression  model  to  predict  cost 
growth  based  on  service  and  find  that  Army  programs  correspond  with  higher  cost 
growth.  However,  if  the  Army  programs  in  the  dataset  tend  to  be  smaller  than  the  Navy 
programs,  this  difference  could  be  problematic,  since  Dameron  et  al.’s  (2002)  study  finds 
that  size  impacts  cost  growth  with  smaller  programs  correlating  with  higher  cost  growth. 
Thus,  the  results  are  unclear  as  to  whether  Army  programs  correspond  with  higher  cost 
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growth  or  smaller  programs  correspond  with  higher  cost  growth.  By  using  multiple 
regression,  one  can  determine  the  relationship  between  service  and  cost  growth  while 
controlling  for  program  size,  thus  resolving  the  dilemma  faced  by  using  simple  linear 
regression. 

Another  benefit  to  multiple  regression  is  that  the  relationship  between  the 
predictor  and  response  variables  is  not  limited  to  a  linear  relationship;  rather,  multiple 
regression  allows  for  non-linear  relationships  (McClave  et  ah,  2005).  For  example,  if  the 
relationship  between  cost  growth  and  years  of  development  increases  initially  and  then 
decreases  after  a  certain  point,  a  quadratic  model  would  be  more  appropriate  (Figure  10). 


Figure  10.  Quadratic  Relationship 

In  addition  to  simple  and  multiple  regression,  a  series  of  AFIT  theses,  beginning 
with  Sipple  (2002),  also  includes  logistic  regression  in  order  to  create  models  in  which 
the  response  variable  is  binary  (that  is,  the  response  has  only  two  possible  values).  In  the 
case  of  these  AFIT  theses,  the  analyses  use  a  two-stage  process  consisting  of  logistic  and 
multiple  regression.  First,  logistic  regression  models  are  built  from  the  entire  dataset  in 
order  to  predict  whether  or  not  a  weapon  system  program  is  likely  to  experience  cost  or 
schedule  growth  (response  value  either  “yes”  or  “no”).  Then,  multiple  regression  models 
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are  built  from  the  portion  of  the  dataset  that  experieneed  cost  or  schedule  growth  in  order 
to  predict  how  much  growth  will  occur  (Cross,  2006;  Monaco,  2005;  Genest,  2004; 

Lucas,  2004;  McDaniel,  2004;  Rossetti,  2004;  Bielecki,  2003;  Moore,  2003;  Sipple, 
2002).  If  the  models  were  to  simply  use  multiple  regression  for  the  entire  dataset,  a  large 
portion  of  the  data  points  would  have  values  of  zero.  This  would  result  in  a  dilution  in 
the  predicted  amount  of  growth  as  well  as  result  in  violation  of  some  assumptions 
required  for  regression  analyses,  such  as  normality  and  constant  variance  for  the  residuals 
(the  difference  between  the  predicted  and  actual  value)  (Bielecki  and  White,  2005).  See 
Chapters  III  and  IV  for  a  more  detailed  discussion  of  logistic  regression  and  residual 
analyses. 

Chapter  Summary 

This  chapter  has  reviewed  previous  cost  and  schedule  growth  studies  examining 
all  defense  programs,  as  well  as  those  that  focus  exclusively  on  space  systems.  The 
literature  review  reveals  that  space  system  cost  and  schedule  growth  studies  have  been 
sparse  and  primarily  qualitative,  while  quantitative  studies  analyzing  cost  and  schedule 
growth  for  all  defense  programs  have  been  numerous.  By  reviewing  the  literature,  this 
chapter  has  laid  the  foundation  for  answering  four  of  the  investigative  questions: 

1 .  Which  systems  should  be  considered  “space  systems”? 

2.  What  are  the  current  methods  for  calculating  cost  and  schedule  growth? 

3.  What  characteristics  of  the  program  or  acquisition  environment  are  good 
predictors  of  cost  and  schedule  growth? 

4.  What  are  the  current  methodologies  for  predicting  cost  and  schedule  growth? 
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With  respect  to  Investigative  Question  1 ,  the  literature  reveals  several  definitions  for 
“space  system.”  The  most  restrictive  definitions  consider  “space  systems”  to  be  only 
those  systems  operating  in  the  space  environment,  such  as  satellites.  Broader  definitions 
can  include  launch  vehicles,  space  system  ground  equipment,  and  even  strategic  missiles. 
Limiting  “space  systems”  to  include  only  those  operating  in  the  space  environment  has 
the  advantage  of  ensuring  that  one  is  comparing  similar  systems  (“apples  to  apples”). 
However,  by  using  the  restrictive  definition,  researchers  do  not  capture  the  entire  system 
necessary  for  executing  space  missions.  If  the  focus  is  on  the  DoD  space  acquisition 
community,  the  restrictive  definition  does  not  capture  all  of  the  systems  this  community 
acquires  since  the  DoD  space  acquisition  community  is  responsible  for  acquiring  not  only 
space-based,  but  also  land-based  assets  including  strategic  missiles. 

When  exploring  Investigative  Question  2,  cost  and  schedule  growth  studies 
provide  two  methods  for  calculating  growth,  as  a  percentage  or  as  a  growth  factor.  For 
cost  growth,  the  major  differences  in  the  calculations  are  not  in  the  formulas,  but  rather  in 
the  definition  of  actual  and  estimated  costs.  These  differences  include:  how  far  along  in 
development  a  program  needs  to  be,  which  variances  to  include,  and  how  to  adjust  for 
quantity. 

Investigative  Question  3  provides  valuable  insight  into  predictors  of  cost  and 
schedule  growth.  Interestingly,  for  cost  growth,  the  qualitative  studies  and  the 
quantitative  studies  differ  on  the  factors  they  considered  and  thus  differ  on  which  factors 
they  find  contribute  most  to  cost  growth.  The  most  likely  cause  of  this  disconnect  is  that 
quantitative  studies  often  limit  their  predictor  variables  to  those  available  in  the  SAR,  and 
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many  of  the  factors  considered  in  qualitative  studies  are  unlikely  to  be  available  for  the 
large  number  of  programs  considered  by  quantitative  studies. 

With  respect  to  Investigative  Question  4,  the  literature  demonstrates  a  range  of 
qualitative  and  quantitative  methodologies  for  assessing  the  predictors,  with  each  method 
having  strengths  and  limitations.  Qualitative  methods,  such  as  case  studies,  provide  the 
advantage  of  a  depth  of  understanding  into  the  programs  being  examined,  but  the  small 
sample  size  limits  the  ability  to  generalize  the  results  to  other  systems.  Quantitative 
methods,  such  as  regression  analysis,  provide  the  advantage  of  statistical  rigor  and  enable 
the  researcher  to  numerically  assess  how  much  a  predictor  contributes  to  cost  or  schedule 
growth.  However,  quantitative  methods  may  be  limited  by  available  data,  and  thus 
unable  to  account  for  all  factors  contributing  to  space  system  cost  or  schedule  growth. 

The  literature  reviewed  herein  contributes  to  this  study  by  providing  a  greater 
understanding  of  how  past  researchers  have  defined  space  systems,  calculated  cost  and 
schedule  growth,  identified  predictors,  and  assessed  those  predictors.  This  study  builds 
upon  this  literature  over  the  course  of  the  next  several  chapters.  Chapter  III  presents  this 
study’s  definition  of  “space  system,”  method  for  calculating  growth,  and  methodology 
for  predicting  space  system  cost  and  schedule  growth.  Chapter  IV  incorporates  the 
predictors  identified  by  the  literature  to  quantitatively  assess  which  predictors  are  best  for 
predicting  space  system  cost  and  schedule  growth. 
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III.  Methodology  and  Data 


Chapter  Overview 

This  study  defines  “space  systems”  using  the  broadest  definition:  including  not 
only  satellites  and  spacecraft,  but  also  launch  vehicles,  strategic  missiles,  and  space- 
related  ground  equipment.  This  study  uses  two  separate  sets  of  data:  Department  of 
Defense  (DoD)  data  and  National  Aeronautics  and  Space  Administration  (NASA)  data. 
The  DoD  dataset  includes  satellites,  launch  vehicles,  strategic  missiles,  and  space-related 
ground  equipment;  the  NASA  dataset  only  includes  spacecraft  and  satellites.  This 
chapter  begins  with  the  discussion  of  the  methodology  used  by  this  study,  and  concludes 
with  a  discussion  of  each  of  the  datasets.  Each  dataset  discussion  details  the  data  source, 
the  response  variables  and  how  they  are  calculated,  the  potential  predictor  variables,  and 
the  diagnostics  used  to  validate  the  models. 

Methodology 

Figure  1 1  presents  the  methodology  used  in  this  study.  The  methodology  begins 
with  collection  of  the  data,  proceeds  to  a  preliminary  and  inferential  analysis  of  the  data, 
and  concludes  with  interpreting  the  results. 

Data  Collection 

The  literature  review  aids  the  data  collection  process  by  identifying  potential 
predictor  variables,  as  well  as  sources  of  data  for  previous  studies.  In  the  case  of  this 
study,  two  separate  datasets  are  compiled  -  one  for  DoD  space  systems  and  one  for 
NASA  space  systems.  These  datasets  are  kept  separate  due  to  the  differences  in  the  types 
of  space  systems  included  and  the  available  data  for  the  predictor  variables. 


46 


Figure  11.  Methodology  Flow  Chart 
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The  primary  source  of  data  for  DoD  space  systems  is  the  Selected  Acquisition 
Report  (SAR)  (Selected,  2003;  “Selected,”  2007;  “DAMIR,”  2007).  As  discussed  in 
Chapter  II,  the  SAR  includes  cost,  schedule,  and  programmatic  information  for  Major 
Defense  Acquisition  Programs.  The  DoD  dataset  consists  of  data  for  21  programs 
including  satellites,  launch  vehicles,  missiles,  and  ground  equipment.  See  DoD  data 
discussion  herein  for  more  detailed  information  regarding  the  data  used  in  this  study. 

The  NASA  dataset  combines  data  from  two  previous  cost  growth  studies:  the 
1992  Institute  for  Defense  Analyses  study  (Tyson  et  ah,  1992a)  and  the  2004  NASA 
Headquarters  Cost  Analysis  Division  study  (Schaffer,  2004).  Additional  data  were  added 
to  this  dataset  based  on  publicly  available  online  NASA  sources  including  National  Space 
Science  Data  Center  (“NSSDC,”  2007),  JPL  Mission  and  Space  Craft  Library  (“MSL,” 
2007),  and  NASA’s  Science  Mission  Directorate  (“Science,”  2007).  Personal 
communications  with  program  personnel  provided  additional  data.  The  NASA  dataset 
includes  cost,  schedule,  and  descriptive  data  for  71  satellites  and  spacecraft.  The 
availability  of  schedule  data  for  47  of  these  systems  allows  for  this  study  to  include  an 
analysis  of  schedule  growth  in  addition  to  cost  growth.  See  NASA  data  discussion  herein 
for  more  detailed  information  regarding  the  data  used  in  this  study. 

After  compiling  the  data  into  their  respective  datasets,  the  data  go  through  a 
rigorous  “data  scrub.”  This  data  scrub  includes  analyses  on  individual  variables  to  ensure 
that  all  of  the  values  have  been  entered  correctly  and  all  calculated  fields  are  correctly 
tabulated.  Additionally,  the  data  scrub  identifies  any  unusual  patterns  or  observations 
within  a  particular  variable.  For  example.  Figure  12  displays  a  histogram  of  the  cost 
growth  values  for  the  DoD  space  systems.  As  can  be  seen,  there  is  an  outlier  that  has 
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three  times  the  cost  growth  of  the  next  highest  observation.  This  observation  is  noted  and 
may  be  removed  later  on  if  it  unduly  influences  the  models. 
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Figure  12.  Histogram  of  DoD  Cost  Growth 


In  addition  to  identifying  single  points  that  are  unusual,  the  data  scrub  also  identifies 
unusual  patterns,  such  as  distributions  that  appear  bimodal''  rather  than  normal.  Unusual 
patterns  may  indicate  the  need  to  create  additional  variables  to  capture  these  patterns. 

See  the  discussion  herein  on  the  NASA  response  variables,  for  details  on  creating  logistic 
response  variables  to  capture  bimodal  distributions. 

Preliminary  Analysis 

The  cost  and  schedule  growth  analysis  begins  with  the  use  of  graphical 
descriptive  methods.  These  methods  compare  the  cost/schedule  growth  response  variable 
to  a  single  predictor  variable  using  bar  charts  to  compare  means.  After  identifying 
potential  patterns,  the  Preliminary  Analysis  uses  Student’s  t-tests  to  determine  the 
statistical  significance  of  these  relationships.  Correlation  analyses  are  also  used  to 

*’  Bimodal  distributions  have  two  distinct  modes  (that  is,  two  relative  maximum  values).  Such  a 
distribution  may  represent  two  separate  distributions. 
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identify  relationships  between  predictor  variables.  See  Chapter  IV  for  the  Preliminary 
Analysis  and  more  information  on  the  analytical  techniques  used. 

Inferential  Analysis 

Simple  Regressions.  The  inferential  statistical  analysis  begins  with  performing 
simple  regressions.  For  each  response  variable,  each  predictor  variable  is  regressed 
against  it  individually  to  determine  which  predictors  are  the  best  indicators  of  cost  or 
schedule  growth.  This  consists  of  several  dozen  regressions  for  each  response  variable. 

Preliminary  Model.  Those  predictor  variables  that  are  significant  at  the  5%  level 
represent  the  main  drivers  for  that  response  variable  and  are  carried  forward  to  the 
preliminary  model.  Based  on  the  correlation  analysis  from  the  Preliminary  Analysis,  if 
two  predictors  are  highly  correlated,  then  only  the  predictor  that  was  more  predictive  (in 
terms  of  significance  level)  is  carried  forward.  The  preliminary  model  combines  all  non- 
correlated  significant  predictor  variables  into  a  single  model. 

Preliminary  Model  Diagnostics.  In  order  to  validate  the  models,  the  preliminary 
model  undergoes  numerous  diagnostic  tests.  For  linear  regression  models,  this  includes 
comparing  the  R  and  adjusted  R  ,  using  Cook’s  Distance  to  identify  influential  data 
points,  testing  the  studentized  residuals  for  normality  using  the  Shapiro-Wilk  Test,  and 
testing  the  residuals  for  constant  variance  using  the  Breusch-Pagan  test.  For  logistic 
models,  diagnostics  include  assessing  the  R  (U),  studying  the  ROC  Curve,  and 
comparing  the  Wald  and  Likelihood  Ratio  Parameter  Estimates  to  identify  unstable 
variables.  If  the  diagnostics  reveal  that  the  model  is  sound,  then  the  model  proceeds  to 
the  refinement  stage.  If  the  diagnostics  reveal  that  the  model  is  not  sound,  the  individual 
variables  are  revisited  for  adjustment.  This  could  include  removal  of  influential  data 
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points,  transformations  (such  as  logarithmic)  of  the  response  or  predietor  variables, 
identifieation  of  new  predictor  variables,  or  transforming  a  variable  from  a  eontinuous 
variable  to  a  discrete  variable. 

Model  Refinement.  In  order  to  identify  the  most  predictive  model(s),  the 
preliminary  model  undergoes  an  iterative  proeess  of  running  hundreds  of  regressions  by 
adding  and  removing  the  remaining  predictor  variables  individually  and  in  groups  to 
determine  if  the  addition  or  removal  of  such  variables  adds  value  to  the  model. 
Refinement  also  includes  testing  for  interaetion  terms  and  higher-order  terms,  such  as 
quadratics.  These  new  models  are  evaluated  against  the  preliminary  model  and  eaeh 
other  by  comparing  the  and  adjusted  R^  for  linear  regression  models  and  by  eomparing 
the  R  (U)  and  using  the  Likelihood  Ratio  tests  for  logistie  models. 

Refined  Model  Diagnostics.  In  order  to  validate  the  refined  models,  these  models 
undergo  the  same  diagnosties  as  the  preliminary  models.  If  the  diagnosties  reveal  that  the 
model  is  theoretieally  sound,  the  most  predietive  model(s)  are  established  as  the  final 
model(s).  Otherwise,  the  data  or  methodology  goes  through  additional  adjustments.  See 
Chapter  IV  for  the  final  models. 

Results 

After  establishing  the  final  models,  the  analysis  proceeds  to  an  interpretation  of 
the  models  and  associated  predietor  variables.  See  Chapter  V  for  the  discussion  of  the 
results,  limitations  to  the  results,  and  reeommendations  for  further  study. 

DoD  Space  Systems  Dataset 

The  DoD  dataset  uses  information  annually  reported  to  Congress  through  the 
SAR.  The  21  spaee  programs  included  in  this  dataset  are  satellites,  launch  vehicles. 
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strategic  missiles,  and  space-related  ground  equipment  reported  in  SARs  between  1969 
and  2006  (see  Appendix  A  for  a  complete  list  of  the  space  systems).  The  dataset  includes 
total  costs  of  all  variance  categories  (with  the  adjustments  described  herein)  for  both 
development  and  procurement  costs  associated  with  the  development  phase  of  system 
acquisition  (Phase  B:  Design  Phase,  see  Figures  3  and  4  in  Chapter  I).  Like  most  studies 
using  SAR  data,  this  study  uses  a  mix  of  completed  and  on-going  programs.  To  ensure 
enough  cost  data  were  available  for  each  program,  this  study  follows  the  example  of 
McNichol  (2005)  by  setting  a  minimum  requirement  that  a  program  had  to  have  reported 
SARs  for  at  least  three  years  in  order  to  qualify  for  inclusion. 

It  is  important  to  recognize  that  using  SAR  data  has  a  number  of  drawbacks, 
primarily  due  to  the  nature  of  the  reporting  process.  Hough  (1992)  notes  a  number  of 
limitations  from  using  data  from  the  SAR  including: 

•  Programs  do  not  always  use  a  consistent  baseline  for  the  cost  estimates 

•  Not  all  elements  of  cost  are  included 

•  Certain  classes  of  programs,  such  as  special  access  programs,  are  not  included 

•  Guidelines  for  preparing  SARs  change  over  time 

•  Differences  exist  in  the  interpretation  of  preparation  guidelines 

•  Some  programs  account  for  risk  by  including  reserve  funds  in  cost  estimates 

•  Inconsistencies  exist  in  reporting  for  programs  that  share  costs  between  services 

•  Cost  changes  are  reported  in  terms  of  their  effects  rather  than  root  causes 
While  recognizing  these  limitations,  this  study  chooses  to  use  SAR  data  because  it  does 
provide  a  significant  amount  of  consistency  in  the  type  of  data  that  is  collected  and  the 
format  in  which  that  data  is  available. 


52 


DoD  Response  Variables 

The  DoD  response  variables  include  Total  Cost  Growth  and  Per  Unit  Cost 
Growth.  Due  to  the  inconsistency  in  records  of  schedule  data,  this  analysis  does  not 
include  an  assessment  of  DoD  space  system  schedule  growth. 

Total  Cost  Growth.  The  Total  Cost  Growth  response  variable  compares  the  actual 
or  most  current  estimate  (CE)  to  the  original  development  estimate  (DE)  from  the 
initiation  of  Milestone  B.  The  CE  is  adjusted  for  quantity  changes  by  subtracting  the  cost 
growth  listed  in  the  Quantity  Variance  category.  Both  the  DE  and  CE  are  adjusted  for 
inflation  by  converting  both  into  Constant  Year  2007  (CY07)  dollars.  Cost  Growth  is 
then  calculated  using  Equation  3,  which  provides  Cost  Growth  in  terms  of  a  percentage, 
where  a  value  of  “zero”  means  there  is  no  cost  growth,  a  negative  value  means  that  the 
actual  cost  (CE)  is  less  than  the  planned  costs  (DE),  and  a  positive  value  means  that  the 
actual  cost  (CE)  is  greater  than  the  planned  cost  (DE). 

Cost  Growth  =  (CE-DE)  (3) 

DE 

Since  military  cost  growth  studies  and  current  defense  acquisition  policies  are 
inconsistent  as  to  whether  or  not  strategic  missiles  are  treated  as  space  systems,  this  study 
is  interested  in  analyzing  the  defense  data  both  with  and  without  inclusion  of  these 
systems.  Eigure  13  shows  the  histogram  for  the  21  programs  in  the  DoD  dataset.  The 
histogram  reveals  the  degree  to  which  the  data  for  the  response  variable  represents  a 
normal  distribution.  Although  normality  is  not  required,  tests  on  preliminary  models 
from  this  dataset  reveal  that  the  lack  of  normality  for  this  distribution  leads  to  numerous 
problems  with  the  residual  diagnostics,  and  thus  this  lack  of  normality  prevents  adequate 
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modeling  of  this  variable.  Using  the  Shapiro-Wilk  Test  to  assess  the  normality  of  the 
distribution  yields  a  p-value  of  <0.0001.  Since  this  is  well  below  the  0.05  level  of 
significance,  the  test  rejects  the  null  hypothesis  that  the  data  are  from  a  normal 
distribution.  As  can  be  seen  visually  from  Figure  13,  a  single  point  (Titan  IV)  clearly 
prevents  this  distribution  from  being  a  normal  distribution. 
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Figure  13.  Flistogram  of  DoD  Total  Cost  Growth 


Omitting  this  single  observation  (Figure  14),  improves  the  distribution’s  Shapiro-Wilk 
Test  p-value  to  0.0092;  however,  this  value  is  still  too  low,  thus  rejecting  that  the  data  are 
from  a  normal  distribution.  Even  with  the  removal  of  Titan  IV,  preliminary  models 
continue  to  have  numerous  problems  with  residual  diagnostics.  Thus,  the  data  are  unable 
to  be  sufficiently  analyzed  in  the  current  form. 


54 


Figure  14.  Flistogram  of  DoD  Total  Cost  Growth, 

Titan  IV  omitted 

Since  analysis  of  space  systems  excluding  strategic  missiles  is  also  of  interest,  the 
analysis  tests  the  data  excluding  these  observations  to  see  if  normality  is  still  an  issue. 
Figure  15,  displays  the  histogram  for  Total  Cost  Growth  after  removing  Titan  IV  and  the 
five  strategic  missile  observations.  This  distribution’s  Shapiro-Wilk  Test  p-value  is 
0.0542,  thus  failing  to  reject  that  the  data  are  from  a  normal  distribution.  Although 
borderline,  this  is  a  significant  improvement,  and  this  dataset  of  15  points  is  carried 
forward  into  the  analysis  for  Total  Cost  Growth.  Thus,  in  the  case  of  Total  Cost  Growth, 
the  analysis  focuses  only  on  space  systems  excluding  strategic  missiles. 
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Figure  15.  Flistogram  of  DoD  Total  Cost  Growth, 

Excluding  strategic  missiles  and  Titan  IV 

Per  Unit  Cost  Growth.  Similar  to  the  Total  Cost  Growth  response  variable,  the 

Per  Unit  Cost  Growth  response  variable  compares  the  actual  or  most  current  estimate 

(CE)  to  the  original  development  estimate  (DE),  adjusted  for  inflation  into  CY07  dollars. 

The  Per  Unit  Cost  Growth  response  variable  accounts  for  changes  in  quantity  by 

adjusting  both  the  DE  and  the  CE  into  a  per  unit  cost,  using  Equation  4; 

Per  Unit  =  [(CE/#  of  unit  for  CE)-(DE/#  of  units  for  DE)1  (4) 

Cost  Growth  DE/#  of  units  for  DE 

Eigure  16  displays  the  Per  Unit  Cost  Growth  response  variable  using  all  21  observations 

(includes  strategic  missiles).  Visually,  the  graph  appears  normal,  and  it  has  a  Shapiro- 

Wilk  Test  p-value  of  0.481 1,  well  above  the  0.05  level  of  significance,  thus  failing  to 

reject  that  the  data  are  normally  distributed. 
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Figure  16.  Histogram  of  DoD  Per  Unit  Cost  Growth 


Recognizing  the  utility  of  a  model  that  excludes  strategic  missiles,  the  study  also 
considers  the  data  without  the  inclusion  of  these  systems.  Figure  17  displays  the  Per  Unit 
Cost  Growth  response  variable  excluding  the  strategic  missile  observations.  Visually, 
this  graph  also  appears  normal,  and  it  has  a  Shapiro-Wilk  Test  p-value  of  0.7052,  thus 
failing  to  reject  that  the  data  are  normally  distributed.  Thus,  the  Per  Unit  Cost  Growth 
response  variable  is  carried  forward  for  analysis  using  both  versions  (with  and  without 
strategic  missiles). 
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Figure  17.  Flistogram  of  DoD  Per  Unit  Cost  Growth 
Excluding  strategic  missiles 

DoD  Predictor  Variables 

The  study  assesses  the  predictor  variables  to  determine  which  characteristics  are 
indicators  of  cost  growth.  Some  of  these,  such  as  Commodity  Type,  have  been  used  in 
past  quantitative  cost  growth  studies,  others  such  as  Program  Manager  Tenure  are 
inspired  by  the  qualitative  space  studies,  and  others  such  as  Mission  Type  are  new 
additions. 

Commodity  Type.  This  predictor  captures  the  commodity  classification  of  the 
particular  program.  The  attribute  is  represented  in  the  dataset  by  three  separate  binary 
variables:  Satellite,  Launch  Vehicle/Missile,  and  Ground  Equipment.  In  each  case,  the 
variable  is  assigned  a  value  of  “1”  if  the  system  belongs  to  that  category  and  a  value  of 
“0”  otherwise. 

Mission  Area.  This  attribute  captures  the  type  of  mission  for  the  program  (note 
that  a  program  may  belong  to  more  than  one  mission  area).  As  with  Commodity  Type, 
Mission  Area  is  measured  by  four  separate  binary  variables:  Communications, 
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Navigation,  Earth  Observation  (such  as  visual  or  infrared  scanning),  and  Space  Support 
(those  systems  that  perform  a  significant  portion  of  their  mission  on  land,  such  as  launch 
vehicles,  strategic  missiles,  and  ground  equipment).  In  each  case,  the  variable  is  assigned 
a  value  of  “1”  if  the  system  belongs  to  that  category  and  a  value  of  “0”  otherwise. 

Program  Size.  This  attribute  is  a  continuous  variable  measured  in  terms  of  actual 
system  cost  in  CY07  dollars.  As  discussed  in  the  literature  review,  some  studies  have 
found  that  smaller  programs  are  associated  with  higher  cost  growth. 

Development  Duration.  This  is  a  continuous  variable  measuring  the  number  of 
years  between  the  first  development  estimate  and  the  last  development  estimate  included 
in  the  study.  As  discussed  in  the  literature  review,  studies  have  found  that  those 
programs  with  longer  development  periods  (or  programs  further  along  in  their 
development)  are  associated  with  higher  cost  growth.  Due  to  the  inconsistency  in  records 
of  schedule  data,  it  was  not  possible  to  include  other  schedule  variables. 

Program  Managers.  Qualitative  DoD  space  systems  studies  identified  high 
rotation  of  Program  Managers  as  a  contributor  to  space  system  cost  growth.  Thus,  two 
Program  Manager  variables  have  been  included:  #  of  PMs  and  PM  Tenure.  #  of  PMs  is  a 
discrete  variable  measuring  the  number  of  Program  Managers  during  the  system’s 
development  phase,  while  PM  Tenure  is  a  continuous  variable  measuring  the  average 
tenure  of  a  Program  Manager,  calculated  by  dividing  the  Development  Duration  by  the  # 
of  PMs. 

Baselines.  Adjusting  the  baseline  is  an  indication  of  major  program  restructuring. 
Thus,  two  baseline  variables  have  been  included  to  attempt  to  capture  major 
programmatic  changes.  The  first,  #  of  Baselines  is  a  discrete  variable  measuring  the 
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number  of  baselines  for  the  system.  The  seeond,  Baselines/yr  is  a  eontinuous  variable 
measuring  the  number  of  baselines  adjusted  for  length  of  the  development,  ealeulated  by 
dividing  the  #  of  Baselines  by  the  Development  Duration. 

Contract  Type.  This  attribute  eaptures  the  type  of  eontraets  that  were  used  in  the 
development  of  the  program.  It  is  measured  by  two  separate  binary  variables:  Cost  Plus 
Award  Fee  (CPAF)  and  Firm  Fixed  Price  (FFP).  In  eaeh  ease,  the  variable  is  assigned  a 
value  of  “1”  if  the  system  belongs  to  that  eategory  and  a  value  of  “0”  otherwise.  Note 
that  there  are  other  eontraet  types  available,  but  these  two  are  the  most  widely  used  for 
the  programs  in  the  DoD  dataset.  Many  programs  use  multiple  eontraet  types  for  various 
portions  of  development.  Thus,  it  is  feasible  for  a  program  to  have  both  a  CPAF  and  a 
FFP  eontraet  or  to  have  neither. 

Lead  Service:  Air  Force.  This  is  a  binary  variable  with  a  value  of  “1”  if  the 
system  was  developed  by  the  Air  Foree  and  a  value  of  “0”  otherwise. 

Cost  Breach.  This  is  a  binary  variable  with  a  value  of  “1”  if  the  system 
experieneed  a  eost  breaeh  [eost  exeeeded  10%  of  objeetive  eost  reported  in  the 
Acquisition  Program  Baseline  (APB)  (Axtell  and  Irby,  2007)]  during  development  and  a 
value  of  “0”  otherwise. 

Schedule  Breach.  This  is  a  binary  variable  with  a  value  of  “1”  if  the  system 
experienced  a  schedule  breach  [schedule  exceeded  6  months  from  objective  schedule 
reported  in  the  APB  (Axtell  and  Irby,  2007)]  during  development  and  a  value  of  “0” 
otherwise. 

Prime  Contractor.  This  attribute  is  measured  by  four  separate  binary  variables: 
Lockheed  Martin,  Boeing,  Northrop  Grumman,  and  Other  Contractor.  In  each  case,  the 


60 


variable  is  assigned  a  value  of  “1”  if  the  system  was  developed  by  that  contractor  (or  one 
of  that  contractor’s  predecessors)  and  a  value  of  “0”  otherwise.  Note  that  in  some  cases, 
a  system  may  have  multiple  prime  contractors. 

NASA  Space  Systems  Dataset 

The  NASA  dataset  includes  data  compiled  from  the  1992  Institute  for  Defense 
Analyses  study  (Tyson  et  ah,  1992a);  the  2004  NASA  Headquarters  Cost  Analysis 
Division  study  (Schaffer,  2004);  publicly  available  online  NASA  sources  including 
National  Space  Science  Data  Center  (“NSSDC,”  2007),  JPL  Mission  and  Space  Craft 
Library  (“MSL,”  2007),  and  NASA’s  Science  Mission  Directorate  (“Science,”  2007);  as 
well  as  data  collected  through  personal  communications  with  program  personnel.  The 
NASA  dataset  includes  cost,  schedule,  and  descriptive  data  for  71  satellites  and 
spacecraft  from  1964  to  2004.  The  cost  data  includes  total  development  costs  through  the 
launch  of  the  spacecraft.  Unlike  the  DoD  dataset  which  includes  both  completed  and  on¬ 
going  programs,  all  71  NASA  programs  have  completed  development  and  been  launched. 
See  Appendix  B  for  a  complete  list  of  the  NASA  programs  used  in  this  study.  Since 
initial  launch  estimates  and  actual  launch  dates  were  available  for  47  of  the  programs  in 
the  NASA  dataset,  the  response  variables  include  both  cost  and  schedule  growth. 

NASA  Response  Variables 

NASA  Cost  Growth.  Similar  to  the  DoD  cost  growth  response  variables,  the 
NASA  Cost  Growth  response  variable  compares  actual  development  costs  to  the  initial 
estimate  in  terms  of  a  percentage,  using  Equation  5.  The  estimate  and  actual  costs  are 
adjusted  for  inflation  by  converting  both  into  Constant  Year  2007  (CY07)  dollars. 
Because  NASA  programs  tend  to  be  formulated  around  the  development  of  a  single 
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system,  with  each  spacecraft  considered  a  separate  program,  quantity  adjustments  are  not 
required. 


Cost  Growth  =  ( Actual-Estimate)  (5) 

Estimate 

The  histogram  (Figure  18)  of  NASA  Cost  Growth  reveals  that  a  number  of  data 
points  fall  at  the  high  end  of  the  distribution,  which  may  cause  the  data  to  not  represent  a 
normal  distribution.  Testing  for  normality  using  the  Shapiro-Wilk  Test,  provides  a  p- 
value  of  <0.0001 .  Since  p-values  less  than  0.05  lead  to  the  rejection  of  the  null 
hypothesis  that  the  data  is  from  a  normal  distribution,  it  is  suspected  that  the  values  at  the 
high  end  of  the  NASA  Cost  Growth  data  represent  a  separate  distribution. 


Figure  18.  Histogram  of  Cost  Growth 
response  variable  for  NASA  systems 


Thus,  the  NASA  Cost  Growth  distribution  is  divided  into  two  distributions:  low 
Cost  Growth  (consisting  of  62  observations)  and  high  Cost  Growth  (consisting  of  9 
observations).  After  removing  the  high  cost  growth  programs,  the  histogram  for  low 
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Cost  Growth  (Figure  19)  reveals  a  normal  distribution  with  a  Shapiro-Wilk  Test  p-value 
of  0.7143.  Sinee  this  is  greater  than  0.05,  the  Shapiro-Wilk  Test  fails  to  reject  the  null 
hypothesis  that  the  data  is  from  the  normal  distribution. 


Figure  19.  Flistogram  of  low  Cost  Growth 
response  variable  for  NASA  systems 

Due  to  the  small  sample  size  {n  =  9),  it  is  unclear  from  a  visual  examination  of  the 
histogram  for  high  Cost  Growth  (Figure  20)  whether  these  values  represent  a  normal 
distribution.  Fortunately,  the  Shapiro-Wilk  Test  was  designed  for  small  sample  sizes; 
Shapiro  and  Wilk  (1965)  originally  designed  the  test  for  2  <  n  <  50,  and  the  test  has  been 
shown  to  be  robust  for  samples  n  <  2000  (“JMP®”,  2005;  Arthur  and  Seber,  1984). 
Testing  the  distribution  of  the  high  Cost  Growth  programs  for  normality  results  in  a 
Shapiro-Wilk  p-value  of  0.3468;  since  this  is  greater  than  0.05,  the  Shapiro-Wilk  Test 
fails  to  reject  the  null  hypothesis  that  the  data  is  from  the  normal  distribution.  Given  that 
the  full  NASA  Cost  Growth  distribution  did  not  pass  the  test  for  normality,  but  the 
division  of  it  provides  two  distributions  that  do  pass  the  test  for  normality,  it  is  concluded 


63 


that  the  NASA  Cost  Growth  response  represents  a  bimodal  distribution.  Thus,  in  order  to 
model  cost  growth  for  NASA  space  systems,  it  is  best  to  separately  model  low  Cost 
Growth  and  high  Cost  Growth. 


Figure  20.  Histogram  of  high  Cost  Growth 
response  variable  for  NASA  systems 

Adapting  from  the  technique  implemented  by  Sipple  (2002),  this  study  employs 
logistic  regression  to  determine  whether  or  not  a  program  is  likely  to  experience  high  cost 
growth  using  the  binary  variable  High  Cost  Growth?  with  a  value  “1”  for  high  cost 
growth  programs  and  a  value  of  “0”  for  low  cost  growth  programs.  Logistic  regression 
uses  maximum  likelihood  to  estimate  the  parameters  that  best  model  the  data,  creating  a 
likelihood  function  that  expresses  the  probability  (as  a  value  between  0  and  1)  that  the 
independent  variables  predict  the  dependent  variable.  For  our  purposes,  a  probability 
greater  than  or  equal  to  0.5  (50%)  predicts  a  program  will  experience  high  cost  growth, 
and  a  probability  of  less  than  0.5  predicts  the  program  will  not  experience  high  cost 
growth.  See  Sipple  (2002)  for  more  information  on  the  logistic  response  function. 
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Thus,  the  first  response  variable,  High  Cost  Growth?,  is  a  binary  variable 
measuring  the  likelihood  that  a  program  will  experienee  high  eost  growth.  The  seeond 
response  variable.  Cost  Growth,  is  a  eontinuous  variable  that  measures  the  pereentage  of 
eost  growth  that  a  program  is  likely  to  ineur.  Beeause  the  NASA  Cost  Growth  response 
represents  a  bimodal  distribution,  the  analysis  models  this  variable  twiee,  once  for  each 
distribution,  thus  providing  a  High  Cost  Growth  Linear  Regression  Model  and  a  Low 
Cost  Growth  Linear  Regression  Model. 

Schedule  Growth.  The  Schedule  Growth  response  variable  compares  the  planned 
launch  date  to  the  actual  launch  date  for  47  NASA  space  systems.  Both  dates  are 
measured  in  the  number  of  months  they  occur  from  program  initiation.  As  with  Cost 
Growth,  Schedule  Growth  is  calculated  as  a  percentage  using  Equation  6. 

Schedule  Growth  =  (Actual  Launch  Schedule  -  Planned  Launch  Schedule)  (6) 

Planned  Launch  Schedule 

Similar  to  Cost  Growth,  the  histogram  of  Schedule  Growth  (Figure  21)  does  not 
appear  to  represent  the  normal  distribution.  The  Shapiro-Wilk  Test  rejects  the  null 
hypothesis  that  the  data  are  from  the  normal  distribution  with  a  p-value  of  <0.0001. 
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Figure  2 1 .  Flistogram  of  Schedule  Growth 
response  variable  for  NASA  systems 


Using  the  same  methodology  as  with  Cost  Growth,  the  Schedule  Growth 
distribution  is  divided  into  two,  to  see  if  it  represents  a  bimodal  distribution.  Figures  22 
and  23  provide  the  histograms  for  low  Schedule  Growth  (36  observations)  and  high 
Schedule  Growth  (1 1  observations),  respectively.  Using  the  Shapiro-Wilk  Test,  both 
distributions  fail  to  reject  the  null  hypothesis  that  the  distributions  represent  a  normal 
distribution,  with  a  p-value  of  0.0842  for  low  Schedule  Growth  and  a  p-value  of  0.0504 
for  high  Schedule  Growth. 
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Figure  22.  Flistogram  of  low  Schedule  Growth 
response  variable  for  NASA  systems 


Figure  23.  Histogram  of  high  Schedule  Growth 
response  variable  for  NASA  systems 


Thus,  this  study  models  NASA  Schedule  Growth  in  the  same  manner  as  NASA 
Cost  Growth,  by  first  using  a  binary  response  variable,  High  Schedule  Growth?,  to 
determine  whether  or  not  the  program  is  likely  to  experience  high  schedule  growth. 
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Then,  the  study  uses  a  eontinuous  response  variable,  Schedule  Growth,  to  separately 
model  high  and  low  Schedule  Growth  programs. 

NASA  Predictor  Variables 

The  study  assesses  a  number  of  predietor  variables  to  determine  whieh 
eharacteristies  are  indieators  of  eost  or  sehedule  growth.  The  predietors  inelude 
programs  size,  program  start,  sehedule  (for  eost  growth  response),  cost  growth  (for 
schedule  growth  response),  mission  area,  international  participation,  developer,  life  span, 
and  mass. 

Program  Size.  There  are  two  program  size  variables:  Initial  Program  Size  and 
Final  Program  Size.  Initial  Program  Size  is  a  continuous  variable  measured  in  terms  of 
the  original  estimate  of  the  system  cost  in  CY07  dollars.  Final  Program  Size  is  a 
continuous  variable  measured  in  terms  of  actual  system  cost  in  CY07  dollars.  As 
discussed  in  the  literature  review,  some  studies  have  found  that  smaller  programs  are 
associated  with  higher  cost  growth. 

Program  Start.  This  is  a  continuous  variable  measured  as  the  number  of  years 
from  1964  (this  year  was  chosen  as  the  baseline  since  it  represents  the  earliest  start  date 
in  the  dataset). 

Schedule  Characteristics.  There  are  three  schedule  predictor  variables:  Estimated 
Time  to  Launch,  Actual  Time  to  Launch,  and  Schedule  Growth.  Estimated  Time  to 
Launch  is  a  continuous  variable  measuring  the  initial  planned  launch  date  in  number  of 
months  from  program  initiation.  Actual  Time  to  Launch  is  a  continuous  variable 
measuring  the  actual  launch  date  in  number  of  months  from  program  initiation.  Schedule 
Growth  is  a  continuous  variable  measuring  the  percent  growth  between  the  estimated  and 
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actual  launch  schedule:  (aetual  date  -  planned  date)/planned  date.  The  sehedule  predietor 
variables  are  used  with  the  eost  growth  response  variables  only. 

Cost  Growth.  %  Cost  Growth  is  a  eontinuous  variable  measuring  the  pereent 
growth  between  the  estimated  and  aetual  eost  (See  Equation  6).  The  %  Cost  Growth 
predietor  variable  is  used  with  the  sehedule  growth  response  variables  only. 

Mission  Area.  This  attribute  eaptures  the  type  of  mission  for  the  program  as 
eatalogued  in  the  National  Spaee  Seienee  Data  Center  database  (“NSSDC,”  2007). 
Mission  area  is  measured  by  ten  binary  variables:  Space  Physics,  Engineering,  Earth 
Science,  Planetary  Science,  Astronomy,  Solar  Physics,  Human  Crew,  Communications, 
Life  Science,  and  Microgravity.  In  eaeh  ease,  the  variable  is  assigned  a  value  of  “1”  if  the 
system  belongs  to  that  eategory  and  a  value  of  “0”  otherwise.  Note  that  a  program  may 
have  more  than  one  mission  area. 

International  Participation.  This  is  a  binary  variable  eapturing  whether  or  not 
eountries  other  than  the  U.S.  partieipated  in  the  seientifie,  teehnieal,  or  design  elements 
of  the  spaeeeraft.  The  variable  is  assigned  a  value  of  “1”  if  the  system  ineluded 
international  partieipation  and  a  value  of  “0”  otherwise. 

Developer.  This  attribute  measures  the  primary  organization  responsible  for 
designing  and  manufaeturing  the  spaeeeraft.  It  is  measured  by  nine  binary  variables: 
NASA,  Jet  Propulsion  Laboratory,  Johns  Hopkins  University,  Lockheed  Martin,  Boeing, 
Northrop  Grumman,  DoD,  International  Developer,  and  Other  Developer.  In  eaeh 


Two  DoD  programs,  DSCS-2  and  SCATHA  are  included  in  the  NASA  dataset.  A  number  of  NASA 
databases  have  included  the  cost,  schedule,  and  technical  data  of  these  programs  due  to  their  similarity  to 
other  NASA  programs. 
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case,  the  variable  is  assigned  a  value  of  “1”  if  the  system  was  developed  by  that 
organization  and  a  value  of  “0”  otherwise. 

Life  Span.  There  are  two  life  span  predictor  variables;  Design  Life  and  Actual 
Life.  Design  Life  is  a  eontinuous  variable  measuring  the  intended  design  life  of  the 
spaeecraft  in  months.  Actual  Life  is  a  eontinuous  variable  measuring  the  actual  life  span, 
or  eurrent  estimate  of  the  life  span  for  programs  still  in  operation,  in  months. 

Mass.  There  are  two  mass  predietor  variables:  Total  Mass  and  Dry  Mass.  Total 
Mass  is  a  continuous  variable  measuring  the  total  mass  of  the  spaceeraft  in  kilograms, 
ineluding  eonsumable  propellants,  at  the  time  of  launeh.  Dry  Mass  is  a  eontinuous 
variable  measuring  the  mass  of  the  spaceeraft  in  kilograms,  exeluding  consumable 
propellants,  at  the  time  of  launeh. 

Chapter  Summary 

This  ehapter  diseussed  the  methodology  and  data  used  by  this  study  to  prediet 
eost  and  schedule  growth  for  space  systems.  The  methodology  ineludes  eollecting  and 
reviewing  the  data,  performing  preliminary  analyses,  performing  inferential  analyses,  and 
then  interpreting  the  results. 

The  DoD  dataset  ineludes  eost  and  programmatie  data  for  21  spaee-related 
programs  from  1969  through  2006,  ineluding  satellites,  launch  vehicles,  strategie 
missiles,  and  ground  equipment.  From  this  dataset,  two  response  variables  have  been 
identified:  Total  Cost  Growth  and  Per  Unit  Cost  Growth.  Additionally,  the  study 
identified  a  number  of  programmatie  eharaeteristies  as  potential  predictors  of  cost 
growth.  Chapter  IV  ineludes  the  Preliminary  Analysis  and  Inferential  Analysis  for  both 
of  these  responses. 
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The  NASA  dataset  includes  cost,  schedule,  and  descriptive  data  for  7 1  satellites 
and  spacecraft  from  1964  to  2004.  From  this  dataset,  two  response  variables  have  been 
identified:  Cost  Growth  and  Schedule  Growth.  Due  to  the  bimodal  distribution  of  these 
two  variables,  the  analysis  begins  by  using  logistic  regression  to  determine  whether  or  not 
the  program  is  likely  to  experience  high  growth  and  then  employs  separate  linear 
regressions  for  high  and  low  growth  to  predict  the  quantity  of  growth.  As  with  the  DoD 
dataset,  the  NASA  dataset  discussion  includes  identifying  programmatic  characteristics 
that  are  potential  predictors  of  cost  and  schedule  growth.  Chapter  IV  includes  the 
Preliminary  Analysis  and  Inferential  Analysis  for  both  of  these  responses. 
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IV.  Analysis 


Chapter  Overview 

As  discussed  in  the  Methodology  seetion  of  Chapter  III,  the  analysis  eonsists  of 
two  segments:  the  Preliminary  Analysis  and  the  Inferential  Analysis.  This  ehapter  begins 
by  detailing  the  Preliminary  Analysis  for  the  Department  of  Defense  (DoD)  data  set, 
followed  by  the  Preliminary  Analysis  for  the  National  Aeronauties  and  Spaee 
Administration  (NASA)  dataset.  The  Preliminary  Analysis  eompares  response  values  for 
individual  predictor  variables,  as  well  as  identifies  potential  relationships  between 
predietor  variables.  The  ehapter  then  details  the  Inferential  Analysis,  beginning  with  the 
DoD  dataset  and  eoneluding  with  the  NASA  dataset.  The  Inferential  Analysis  ineludes 
logistie  and  linear  regression  models  useful  for  predieting  eost  and  sehedule  growth. 
Preliminary  Analysis 

The  Preliminary  Analysis  ineludes  graphieal  analyses.  Student’s  t-tests  (t-test), 
and  eorrelation  analyses  for  seleet  predietor  variables.  The  graphieal  analyses  inelude  bar 
graphs  of  mean  (average)  growth  values  by  predietor  variable  in  order  to  identify 
potential  predictors.  Because  examining  solely  the  mean  values  amongst  groups  ean  be 
misleading,  the  Preliminary  Analysis  ineludes  performing  t-tests  to  determine  if  the 
differenees  in  the  means  are  signifieant.  The  Preliminary  Analysis  also  ineludes 
eorrelation  analyses  between  predietor  variables  to  allow  for  the  identifieation  of 
multieollinearity  issues.  The  diseussion  herein  details  only  those  tests  with  signifieant 
results. 

Multieollinearity  oeeurs  when  two  or  more  predietor  variables  are  intereorreltated.  When  two  highly 
eorrelated  predietors  are  used  in  a  regression  model,  they  ereate  linear  redundaney  in  the  model  and 
diminish  the  aeeuraey  of  the  regression  eoeffieients  (Gujarat,  1995:320-322). 
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Preliminary  Analysis:  DoD 

Total  Cost  Growth.  The  study  performed  preliminary  analyses  for  the  Program 
Manager  Tenure,  Contractor,  Commodity  Type,  Mission  Area,  and  Program  Size 
predictor  variables,  using  all  21  programs  in  the  DoD  dataset. 

Figure  24  displays  average  Total  Cost  Growth  by  prime  contractor:  Lockheed 
Martin,  Boeing,  Northrop  Grumman,  and  other  contractors.  From  the  graph,  it  appears 
that  programs  developed  by  Lockheed  Martin  experience  higher  cost  growth  than  other 
prime  contractors.  Using  a  one-tailed  t-test  to  test  the  alternate  hypothesis  of  unequal 
means  results  in  a  p-value  of  0.0925,  therefore  programs  developed  by  Lockheed  Martin 
do  experience  higher  cost  growth  than  the  other  three  contractor  categories  at  the 
significance  level  of  10%,  but  these  results  are  not  significant  at  the  5%  level. 


Total  Cost  Growth  by  Contractor 


Prime  Contractor 


Figure  24.  DoD  Total  Cost  Growth  by  Prime  Contractor 
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Figure  25  displays  Total  Cost  Growth  by  Commodity  Type.  From  viewing  the 
graph,  it  appears  that  ground  equipment  has  the  lowest  cost  growth  and  launch 
vehicles/missiles  have  the  highest  cost  growth.  A  one-tailed  t-test  for  ground  equipment 
results  in  a  p-value  of  0.0262,  well  below  a  5%  significance  level;  however,  the  one- 
tailed  t-test  for  launch  vehicles/missiles  fails  to  reject  the  null  hypothesis  of  equal  means 
with  a  p-value  of  0.2063.  Thus,  ground  equipment  systems  have  statistically  significant 
lower  cost  growth  than  other  systems. 


Total  Cost  Growth  by  Commodity  Type 


Commodity  Type 


Figure  25.  DoD  Total  Cost  Growth  by  Commodity  Type 


Figure  26  displays  Total  Cost  Growth  by  Mission  Area.  Visual  examination  leads 
to  the  alternate  hypothesis  that  communications  missions  have  lower  cost  growth  than 
other  missions.  A  one-tailed  t-test  for  communications  missions  results  in  a  p-value  of 
0.0447,  thus  rejecting  the  null  hypothesis  of  equal  means.  Thus,  systems  with 
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communications  missions  have  statistically  significant  lower  cost  growth  than  other 


systems. 


Total  Cost  Growth  by  Mission  Area 


Mission  Area 


Figure  26.  DoD  Total  Cost  Growth  by  Mission  Area 


Per  Unit  Cost  Growth.  As  with  Total  Cost  Growth,  the  preliminary  analyses 
assessed  Per  Unit  Cost  Growth  using  all  21  programs  in  the  DoD  dataset.  Figure  27 
displays  Per  Unit  Cost  Growth  by  Commodity  Type.  Similar  to  Total  Cost  Growth,  it 
appears  that  ground  equipment  has  lower  per  unit  cost  growth.  A  one-tailed  t-test  for 
ground  equipment  rejects  the  null  hypothesis  with  a  p-value  of  0.0074,  well  below  a  5% 
or  even  1%  significance  level.  Thus,  ground  equipment  systems  have  statistically 
significant  lower  per  unit  cost  growth  than  other  systems. 
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Per  Unit  Cost  Growth  by  Commodity  Type 
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Figure  27.  DoD  Per  Unit  Cost  Growth  by  Commodity  Type 


Figure  28  displays  Fer  Unit  Cost  Growth  by  Mission  Area.  Visual  examination 
leads  to  the  alternate  hypothesis  that  earth  observation  missions  have  higher  per  unit  cost 
growth  than  other  missions.  However,  a  one-tailed  t-test  for  earth  observation  missions 
results  in  a  p-value  of  0.1089,  just  barely  failing  to  reject  the  null  hypothesis  at  the  10% 
level. 
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Per  Unit  Cost  Growth  by  Mission  Area 
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Figure  28.  DoD  Per  Unit  Cost  Growth  by  Mission  Area 


Predictor  Correlations.  The  Preliminary  Analysis  also  includes  performing 
correlation  analysis  on  the  predictor  variables  in  the  DoD  dataset  in  order  to  identify  any 
variables  that  may  exhibit  similar  behaviors,  and  thus  lead  to  multicollinearity  issues.  If 
two  predictors  are  highly  correlated,  using  both  in  a  model  can  create  linear  redundancy 
and  diminish  the  accuracy  of  the  regression  coefficients  (Gujarat,  1995:320-322). 
Franzblau  (1958)  identifies  correlations  values  between  0.60  and  0.80  (or  between  -0.60 
and  -0.80)  as  having  a  “marked  degree  of  correlation”  and  values  between  0.80  and  1.00 
(or  between  -0.80  and  -1.00)  as  having  a  “high  correlation.”  Table  3  provides  correlation 
values  for  those  predictor  variables  that  have  correlations  greater  than  0.6  or  less  than 
-0.6. 
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Tables.  Correlations  between  Do! 

D  Predictor  Varial 

bles 

#  of  PMs 

Development 

Duration 

Cost  Breach 

#  of  Baselines 

Launch 

Vehicle/ 

Missile 

Satellite 

Navigation 

Space  Support 

#  of  PMs 

1.00 

0.92 

0.73 

0.86 

0.65 

Development 

Duration 

1.00 

0.71 

0.82 

0.63 

Cost  Breach 

1.00 

0.66 

#  of  Baselines 

1.00 

0.73 

Launch 

Vehicle/Missile 

1.00 

-0.60 

Satellite 

1.00 

-1.00 

Navigation 

1.00 

Space  Support 

1.00 

In  the  case  of  the  #  of  Program  Managers,  Development  Duration,  Cost  Breach, 
and  #  of  Baselines,  time  is  most  likely  the  underlying  factor  that  ties  these  variables 
together;  that  is,  programs  that  have  had  longer  development  periods  would  be  expected 
to  have  higher  values  for  each  of  these.  During  the  regression  analysis,  if  two  highly 
correlated  variables  were  to  both  appear  significant,  then  including  both  would  cause 
linear  redundancy  in  the  analysis.  In  this  case,  the  variable  that  is  most  predictive  (in 
terms  of  significance  level)  is  kept  in  the  model. 
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Preliminary  Analysis:  NASA 

The  study  includes  preliminary  analyses  for  cost  and  schedule  growth  with  the 
Mission  Area,  Developer,  Program  Size,  and  spacecraft  variables.  The  cost 
analyses  used  all  7 1  programs  in  which  cost  data  were  available,  and  the  schedule 
analyses  used  all  47  programs  in  which  schedule  data  were  available.  The  discussion 
herein  details  those  tests  with  significant  results. 

NASA  Cost  Growth.  Figure  29  displays  average  Cost  Growth  by  Developer. 

Note  that  while  the  average  cost  growth  for  DoD  programs  is  far  greater  than  for  the 
other  developer  categories,  this  dataset  only  includes  two  DoD  programs.  A  one-tailed  t- 
test  for  the  alternate  hypothesis  that  DoD  programs  have  higher  cost  growth  fails  to  reject 
the  null  hypothesis  with  a  p-value  of  0. 1591 .  However,  a  one-tailed  t-test  for  the 
alternate  hypothesis  that  programs  developed  by  Johns  Hopkins  have  lower  cost  growth 
rejects  the  null  hypothesis  with  a  p-value  of  0.0419,  and  a  similar  test  for  NASA 
developed  programs  is  significant  at  the  10%  level  with  a  p-value  of  0.0893.  Thus, 
systems  developed  by  Johns  Hopkins  have  statistically  significant  lower  cost  growth  than 
other  systems  at  the  5%  level,  and  systems  developed  by  NASA  have  statistically 
significant  lower  cost  growth  than  other  systems  at  the  10%  level. 
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Figure  29.  NASA  Cost  Growth  by  Developer 


Schedule  Growth.  Figure  30  displays  average  Schedule  Growth  by  Developer. 
From  this  figure,  it  appears  that  Johns  Hopkins,  Boeing,  and  Other  Developer  are 
associated  with  low  schedule  growth  and  NASA,  Jet  Propulsion  Laboratory,  Northrop 
Grumman,  and  International  Developer  are  associated  with  high  schedule  growth.  One- 
tailed  t-tests  on  these  variables  find  that  these  relationships  are  significant  for;  Johns 
Hopkins  with  a  p-value  of  <0.0001,  Boeing  with  a  p-value  of  0.01 12,  Northrop  Grumman 
with  a  p-value  of  0.0826,  and  Other  Developer  with  a  p-value  of  0.0242.  Thus,  systems 
developed  by  Johns  Flopkins,  Boeing,  and  Other  Developers  have  statistically  significant 
lower  cost  growth  than  other  systems,  and  systems  developed  by  Northrop  Grumman 
have  statistically  significant  higher  cost  growth  than  other  systems. 
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Schedule  Growth  by  Developer 


Figure  30.  NASA  Schedule  Growth  by  Developer 


Predictor  Correlations.  Correlation  analysis  was  preformed  on  the  predictor 
variables  in  the  NASA  dataset  in  order  to  identify  any  variables  that  may  exhibit  similar 
behaviors.  Table  4  provides  only  those  correlations  that  are  greater  than  0.6  or  less  than  - 
0.6.  As  with  the  DoD  predictor  correlations,  if  two  highly  correlated  variables  both 
appear  significant  during  the  regression  analysis,  then  the  variable  that  is  most  predictive 
(in  terms  of  significance  level)  is  kept  in  the  model. 
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Table  4.  Correlations  between  NASA  Predietor  Variables 


Human  Crew 

Life  Science 

Microgravity 

Infl 

Participation 

Infl 

Developer 

Total  Mass 

Dry  Mass 

Human  Crew 

1.00 

0.86 

0.70 

Life  Science 

1.00 

0.81 

Microgravity 

1.00 

Infl  Participation 

1.00 

0.64 

Infl  Developer 

1.00 

Total  Mass 

1.00 

0.99 

Dry  Mass 

1.00 

Inferential  Analysis:  DoD 
Total  Cost  Growth 

As  diseussed  in  Chapter  III,  due  to  the  non-normality  of  the  distribution  when 
ineluding  all  of  the  systems  in  the  DoD  dataset,  the  Total  Cost  Growth  analysis  excludes 
strategic  missiles  and  Titan  IV.  The  remaining  15  systems  include  all  satellites,  other 
launch  vehicles,  and  all  space-related  ground  equipment.  Equation  7  provides  the  final 
model  for  predicting  Total  Cost  Growth  using  the  methodology  outlined  in  Chapter  III; 

CGt  =  0.715  -  0.511* {Communications)  (7) 
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where  CGt  is  the  dependent  variable  and  is  the  predicted  Total  Cost  Growth  as  a 
percentage  of  the  system’s  original  estimate,  and  Communications  is  a  binary  variable 
with  a  value  of  “1”  for  those  systems  with  a  communications  mission  and  a  value  of  “0” 
otherwise.  The  negative  coefficient  on  the  Communications  variable  indicates  that  those 
systems  with  a  communications  mission  experience  lower  cost  growth  than  compared  to 
other  missions.  To  assess  the  ability  of  the  parameter  in  the  model  to  explain  the 

variation  in  the  response,  the  analysis  examines  both  the  R  and  the  adjusted  R  .  The  R 

2  2 

for  this  model  is  0.29  and  the  adjusted  R  is  0.24.  The  relatively  low  values  of  the  R  and 

the  adjusted  R  indicate  that  much  of  the  variation  in  Total  Cost  Growth  is  explained  by 
factors  outside  of  the  model.  Appendix  C  provides  the  complete  output  provided  by  the 
JMP  Software  (JMP®,  2005)  for  this  regression. 

The  analysis  applies  numerous  diagnostics  to  the  model  in  order  to  test  its 
robustness.  The  first  diagnostic  is  the  Cook’s  Distance  test  for  influential  data  points. 
With  this  test,  values  over  0.5  indicate  possible  influential  data  points  (Neter  et  ah, 
1996:381).  The  Cook’s  Distance  for  the  Total  Cost  Growth  model  had  all  points  below 
0.3,  thus  indicating  that  there  are  no  influential  data  points  present.  The  second 
diagnostic  tests  the  studentized  residuals  for  normality  using  the  Shapiro-Wilk  Test. 

With  this  test,  a  p-value  below  0.05  rejects  the  null  hypothesis  that  the  residuals  are 
normally  distributed.  Since  the  optimal  model  will  have  normally  distributed  studentized 


R^and  adjusted  range  from  “0”  to  “1,”  where  a  value  of  “1”  indieates  that  the  parameters  explain 
100%  of  the  variation  of  the  response,  and  a  value  of  “0”  indieates  that  the  parameters  provide  no 
explanation.  Sinee  an  inerease  in  the  number  of  variables  will  result  in  an  inerease  in  the  R^,  the  adjusted 
R^  is  also  refereneed  beeause  it  aeeounts  for  the  number  of  predietor  variables  used  in  the  model.  Thus, 
while  a  saturated  model  (one  with  unneeessary  predietor  variables)  may  have  a  high  R^,  the  saturated  model 
will  not  have  as  high  of  an  adjusted  R^.  Ideally,  the  model  builder  would  want  both  R^  and  adjusted  R^  to 
be  elose  to  “1,”  as  well  as  elose  to  eaeh  other. 
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residuals,  this  requires  Shapiro-Wilk  p-values  over  0.05.  The  Total  Cost  Growth  model 
had  a  Shapiro-Wilk  p-value  of  0.4631;  thus  failing  to  rejeet  the  null  hypothesis.  The  final 
diagnostie  applied  to  the  Total  Cost  Growth  model  tests  the  residuals  for  eonstant 
varianee  using  the  Breuseh-Pagan  Test.  With  this  test,  a  p-value  below  0.05  rejeets  the 
null  hypothesis  that  the  residuals  have  eonstant  varianee.  Thus,  similar  to  the  Shapiro- 
Wilk  Test,  beeause  the  optimal  model  will  have  eonstant  varianee  of  its  residuals,  this 
requires  Breuseh-Pagan  p-values  over  0.05.  The  Breuseh-Pagan  p-value  for  the  Total 
Cost  Growth  model  was  0.1 1 10. 

Per  Unit  Cost  Growth 

The  models  for  predieting  Per  Unit  Cost  Growth  inelude  the  entire  DoD  dataset, 
as  well  as  exeluding  strategie  missiles  in  order  to  provide  a  model  similar  to  the  one 
provided  for  Total  Cost  Growth.  Using  the  methodology  outline  in  Chapter  III,  the  entire 
DoD  dataset  yielded  two  separate  models  for  predieting  Per  Unit  Cost  Growth. 

Equations  8  and  9  provide  these  models: 

CGui  =  0.869  -  0.941^^ {Ground  Equip)  -  0.66U(FFE)  (8) 

where  CGui  is  the  dependent  variable  and  is  the  predieted  Per  Unit  Cost  Growth  as  a 
pereentage  of  the  system’s  original  estimate  for  unit  eost.  Ground  Equip  is  a  binary 
variable  with  a  value  of  “1”  for  ground  equipment  systems  and  a  value  of  “0”  otherwise, 
and  FFP  is  a  binary  variable  with  a  value  of  “1”  for  systems  developed  using  a  Firm 
Fixed  Priee  eontraet  and  a  value  of  “0”  otherwise.  The  negative  eoeffieients  on  both  the 
Ground  Equipment  and  Firm  Fixed  Price  variables  indieate  that  both  of  these  faetors  are 
assoeiated  with  lower  Per  Unit  Cost  Growth.  The  for  this  model  is  0.50  and  the 
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adjusted  R  is  0.44.  Appendix  D  provides  the  eomplete  output  provided  by  the  JMP 
Software  (JMP®,  2005)  for  this  regression. 

The  model’s  diagnostics  are  satisfactory  with  all  points  having  a  Cook’s  Distance 
below  0.15,  the  studentized  residuals  Shapiro-Wilk  p-value  of  0.7392,  and  the  residuals 
Breusch-Pagan  p-value  of  0.0581.  Because  the  Breusch-Pagan  p-value  is  very  close  to 
the  rejection  point  of  being  below  0.05,  the  analysis  examines  the  plot  of  the  residuals 
versus  the  predicted  values  (Figure  31)  to  identify  the  extent  to  which  the  residuals  have 
non-constant  variance.  From  Figure  31,  it  appears  that  a  single  point  to  the  far  left  may 
be  driving  this  low  Breusch-Pagan  p-value.  Removing  this  observation  increases  the 
Breusch-Pagan  p-value  to  0.0721.  Appendix  E  provides  the  complete  output  provided  by 
the  JMP  Software  (JMP®,  2005)  for  this  regression  with  the  single  observation  removed. 
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The  second  model  for  predicting  Per  Unit  Cost  Growth  is: 

CGu2  =  2.069  -  l.nS*{Ground Equip)  -  0.664*(T’M  Tenure)  (9) 

where  CGu2  is  the  dependent  variable  and  is  the  predicted  Per  Unit  Cost  Growth  as  a 
percentage  of  the  system’s  original  estimate  for  unit  cost,  Ground  Equip  is  a  binary 
variable  with  a  value  of  “1”  for  ground  equipment  systems  and  a  value  of  “0”  otherwise, 
and  PM  Tenure  is  a  continuous  variable  representing  the  average  Program  Manager 
Tenure.  The  negative  coefficients  on  both  the  Ground  Equipment  and  PM  Tenure 
variables  indicate  that  both  of  these  factors  are  associated  with  lower  Per  Unit  Cost 
Growth.  The  R  for  this  model  is  0.47  and  the  adjusted  R  is  0.41.  Appendix  F  provides 
the  complete  output  provided  by  the  JMP  Software  (JMP®,  2005)  for  this  regression.  The 
diagnostics  for  this  model  are  all  satisfactory  with  all  points  having  a  Cook’s  Distance 
below  0.15,  the  studentized  residuals  having  a  Shapiro-Wilk  p-value  of  0.3160,  and  the 
residuals  having  a  Breusch-Pagan  p-value  of  0.7838. 

The  last  model  for  Per  Unit  Cost  Growth  removes  the  5  strategic  missile 
observations  from  the  DoD  dataset,  to  provide  a  model  comparable  to  the  one  for  Total 
Cost  Growth.  Equation  10  provides  this  model: 

CGu3  =  0.945  -  l.l53*{Ground Equip)  -  0.666*{FFP)  (10) 

where  CGu3  is  the  dependent  variable  and  is  the  predicted  Per  Unit  Cost  Growth  as  a 
percentage  of  the  system’s  original  estimate  for  unit  cost.  Ground  Equip  is  a  binary 
variable  with  a  value  of  “1”  for  ground  equipment  systems  and  a  value  of  “0”  otherwise, 
and  FFP  is  a  binary  variable  with  a  value  of  “1”  for  systems  developed  using  a  Firm 
Fixed  Price  contract  and  a  value  of  “0”  otherwise.  Note  that  this  model  includes  the  same 
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factors  as  Equation  8,  which  was  for  the  entire  DoD  dataset.  As  with  Equation  8, 
Equation  10  also  has  negative  eoeffieients  for  both  Ground  Equipment  and  Firm  Fixed 
Price,  indieating  that  both  of  these  faetors  are  assoeiated  with  lower  Per  Unit  Cost 
Growth.  The  R  for  this  model  is  0.56  and  the  adjusted  R  is  0.49.  Appendix  G  provides 
the  eomplete  output  provided  by  the  JMP  Software  (JMP®,  2005)  for  this  regression.  The 
diagnosties  for  this  model  are  also  satisfaetory  with  all  points  having  a  Cook’s  Distanee 
below  0.2,  the  studentized  residuals  having  a  Shapiro-Wilk  p-value  of  0.8562,  and  the 
residuals  having  a  Breuseh-Pagan  p-value  of  0.0697. 

Inferential  Analysis:  NASA 

Cost  Growth 

As  diseussed  in  Chapter  111,  due  to  the  bimodal  nature  of  the  distribution, 
modeling  NASA  Cost  Growth  eonsists  of  a  two  stage  proeess.  The  first  stage  ineludes  a 
logistie  regression  model  to  determine  whether  a  program  is  likely  to  experienee  high 
eost  growth.  The  seeond  stage  ineludes  separate  linear  regression  models  for  both  high 
and  low  eost  growth  to  determine  the  likely  pereentage  of  eost  growth.  Note  that  the  low 
eost  growth  model  also  ineludes  zero  and  negative  eost  growth. 

Logistic  Regression  Models.  The  analysis  results  in  two  logistie  models  for 
predieting  the  likelihood  of  a  program  to  experienee  high  eost  growth;  Equations  1 1  and 
13  provide  these  models: 

^  ^  2. 140  -0.05S,*(Initial Program  Size)  +  0.001*(Total Mass) 

g  2.140  -  0.05%* (Initial Program  Size)  +  0.00i*(Total Mass) 

Note  that  JMP®  uses  the  logistic  response  function:  g-dio  +  Bixi  b2X2  +  b3X3)  (“jmP®”,  2005). 

J  _|_  g -(BO  +  BIXI  B2X2  +  B3X3) 

For  ease  of  use  and  interpretation,  the  negative  sign  has  been  multiplied  through  the  parameter  estimates  it 
the  equations  provided  in  the  text.  Thus,  the  parameter  estimates  in  the  JMP®  regression  outputs  in  the 
Appendices  have  the  opposite  sign  as  those  listed  within  this  text. 
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where  Lhcg  is  the  dependent  variable  and  is  a  likelihood  function  that  expresses  the 
probability  that  a  program  will  experience  high  cost  growth,  Initial  Program  Size  is  a 
continuous  variable  that  measures  the  original  estimated  cost  of  the  program  in  CY07 
dollars,  and  Total  Mass  is  a  continuous  variable  that  measures  the  mass  of  the  spacecraft 
(including  consumable  propellants)  in  kilograms.  For  our  purposes,  a  Lhcg  probability 
greater  than  or  equal  to  0.5  (50%)  predicts  a  program  will  experience  high  cost  growth, 
and  a  probability  of  less  than  0.5  predicts  the  program  will  not  experience  high  cost 
growth.  Based  on  the  coefficients,  larger  Initial  Program  Sizes  decreases  the  likelihood 
of  experiencing  high  cost  growth;  whereas  more  massive  spacecraft  increases  the 
likelihood  of  experiencing  high  cost  growth.  To  assess  the  utility  of  the  model,  the 
(U)  is  examined.  The  R  (U)  is  a  ratio  of  likelihoods  (Equation  12)  measuring  the 
proportion  of  the  total  uncertainty  attributed  to  the  fitted  model. 

R  (U)  =  -Loglikelihood  for  Difference  between  Reduced  and  Full  Model  (12) 

-Loglikelihood  for  Reduced  Model 

This  ratio  of  likelihoods  compares  the  uncertainty  from  fitting  the  model  to  the 
uncertainty  from  background  effects  to  determine  whether  the  independent  variables  have 
an  effect  on  the  response  variable  (“JMP®”,  2005).  R^  (U)  ranges  between  0  and  1,  with 
higher  values  indicating  a  more  predictive  model.  R  (U)  values  equal  to  or  greater  than 
0.4  are  desirable  (White,  2007).  The  NASA  High  Cost  Growth?  model  provided  in 
Equation  1 1  has  an  R  (U)  of  0.57.  See  Appendix  H  for  the  complete  logistic  regression 
output  provided  by  the  JMP  Software  (JMP®,  2005). 

Although  the  diagnostics  used  for  linear  regression  analysis  are  not  available  for 
logistic  regression,  the  Receiver  Operating  Characteristic  (ROC)  curve  can  be  used  to 
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assess  the  model’s  accuracy.  The  ROC  curve  distinguishes  between  false-positives  and 
true-positives,  or  in  other  words,  how  often  the  model  predicts  a  value  of  “1”  when  the 
actual  value  is  “0”  compared  to  predicting  a  value  of  “1”  when  the  actual  value  is  “1.”  A 
ROC  curve  that  runs  along  the  45  degree  diagonal  of  the  graph  would  have  an  area  under 
the  curve  of  0.50,  and  would  have  no  predictive  capability.  A  ROC  curve  consisting  of  a 
vertical  line  from  the  point  (0,  0)  to  (0,  1)  and  then  a  horizontal  line  from  (0,  1)  to  (1,  1) 
would  be  perfectly  predictive  and  have  an  area  of  1 .0.  The  logistic  model  in  Equation  1 1 
has  a  ROC  curve  area  of  0.95,  indicating  an  estimated  accuracy  of  95%.  See  Figure  32 
for  the  ROC  curve  of  Lhcgi- 
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Equation  13  provides  the  seeond  model  for  predieting  High  Cost  Growth?: 


T  _  0.741  Program  Size)  +  0M\*(Dry  Mass)  +  ^?t.l05*{Microgravity) 

UHCG2  -  _ 

g  0.741  -0.03^*(Initial Program  Size) +  0.00l*(Dry  Mass)  +  3?:. 105* (Microgravity) 


where  Lhcg  is  the  dependent  variable  and  is  a  likelihood  funetion  that  expresses  the 
probability  that  a  program  will  experienee  high  eost  growth,  Initial  Program  Size  is  a 
eontinuous  variable  that  measures  the  original  estimated  eost  of  the  program  in  CY07 
dollars,  Dry  Mass  is  a  eontinuous  variable  that  measures  the  mass  of  the  spaeeeraft 
(exeluding  eonsumable  propellants)  in  kilograms,  and  Microgravity  is  a  binary  variable 
with  a  value  of  “1”  for  those  systems  with  a  mierogravity  mission  and  a  value  of  “0” 
otherwise.  Based  on  the  eoeffieients,  larger  Initial  Program  Sizes  deereases  the 
likelihood  of  experieneing  high  eost  growth,  more  massive  spaeeeraft  inereases  the 
likelihood  of  experieneing  high  eost  growth,  and  a  mierogravity  mission  inereases  the 
likelihood  of  experieneing  high  eost  growth.  The  (U)  assoeiated  with  this  model  is 
0.50,  and  the  ROC  curve  area  (Figure  33)  is  0.94.  See  Appendix  1  for  the  complete 
logistic  regression  output  provided  by  the  JMP  Software  (JMP®,  2005). 
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Figure  33.  Lhcg2  Receiver  Operating  Curve 


Linear  Regression  Models.  The  analysis  developed  separate  linear  regression 
models  for  high  and  low  cost  growth  programs.  These  models  are  designed  to  be  used  in 
conjunction  with  the  logistic  regression  models  provided  in  Equations  1 1  and  13.  If  the 
logistic  regression  models  predict  that  high  cost  growth  is  likely  to  occur,  then  the  High 
Cost  Growth  Linear  Regression  Model  (Equation  14)  can  be  used  to  predict  the  likely 
percent  cost  growth.  Similarly,  if  the  logistic  regression  models  predict  that  high  cost 
growth  is  not  likely  to  occur,  the  Low  Cost  Growth  Linear  Regression  Model  (Equation 
15)  can  be  used  to  predict  the  likely  percent  cost  growth. 

The  High  Cost  Growth  Linear  Regression  Model  is; 

CGh  =  1.232  +  \.01>l^{Space  Physics)  (14) 
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where  CGh  is  the  dependent  variable  and  is  the  predicted  Cost  Growth  as  a  percentage  of 
the  original  cost  estimate,  and  Space  Physics  is  a  binary  variable  with  a  value  of  “1”  for 
systems  with  a  space  physics  mission  and  a  value  of  “0”  otherwise.  The  positive 
coefficient  on  Space  Physics  indicates  that  space  physics  missions  are  associated  with 
higher  Cost  Growth.  The  for  this  model  is  0.77  and  the  adjusted  is  0.74.  Appendix 
J  provides  the  complete  output  provided  by  the  JMP  Software  (JMP®,  2005)  for  this 
regression.  The  diagnostics  for  this  model  are  all  satisfactory  with  all  points  having  a 
Cook’s  Distance  below  0.4,  the  studentized  residuals  having  a  Shapiro-Wilk  p-value  of 
0.5716,  and  the  residuals  having  a  Breusch-Pagan  p-value  of  0.9189. 

The  Low  Cost  Growth  Linear  Regression  Model  is: 

CGl  =  0.509  -  0.01 4 *(Rrogram  Start)  (15) 

where  CGl  is  the  dependent  variable  and  is  the  predicted  Cost  Growth  as  a  percentage  of 
the  original  cost  estimate,  and  Program  Start  is  a  continuous  variable  measured  as  the 
number  of  years  from  1964.  The  negative  coefficient  on  Program  Start  signifies  that 
more  recent  programs  are  associated  with  lower  cost  growth.  The  R  for  this  model  is 
0.18  and  the  adjusted  R^  is  0.16.  These  low  R^  values  indicate  that  most  of  the  variation 
for  cost  growth  for  systems  that  experience  low  cost  growth  is  due  to  factors  not 
explained  by  the  model.  See  Appendix  K  for  the  complete  output  provided  by  the  JMP 
Software  (JMP®,  2005)  for  this  regression.  The  diagnostics  for  this  model  are  all 
satisfactory  with  all  points  having  a  Cook’s  Distance  below  0.15,  the  studentized 
residuals  having  a  Shapiro-Wilk  p-value  of  0.2607,  and  the  residuals  having  a  Breusch- 
Pagan  p-value  of  0.0841. 
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Schedule  Growth 


As  with  NASA  Cost  Growth,  modeling  NASA  Schedule  Growth  also  consists  of  a 
two  stage  approach:  first  using  logistic  regression  to  predict  the  likelihood  of 
experiencing  high  schedule  growth,  then  using  linear  regression  to  predict  the  quantity  of 
schedule  growth. 

Logistic  Regression  Model.  The  logistic  model  for  High  Schedule  Growth  is 
given  in  Equation  16: 


Lhsg 


-2.186  +  0.00l*(Final Program  Size) 

c _ 

^  -2.186  +  0.00l*(Final Program  Size) 


(16) 


where  Lhsg  is  the  dependent  variable  and  is  a  likelihood  function  that  expresses  the 
probability  that  a  program  will  experience  high  schedule  growth,  and  Final  Program  Size 
is  a  continuous  variable  that  measures  the  final  cost  of  the  program  in  CY07  dollars.  The 
R  (U)  associated  with  this  model  is  0.15,  and  the  ROC  curve  area  (Figure  34)  is  0.76. 
The  relatively  low  R  (U)  and  ROC  curve  areas  indicate  that  this  model  is  not  as 
predictive  as  the  cost  growth  logistic  models.  See  Appendix  L  for  the  complete  output 
provided  by  the  JMP  Software  (JMP®,  2005)  for  this  regression 
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Figure  34.  Lhsg  Receiver  Operating  Curve 


Linear  Regression  Models.  The  analysis  developed  separate  linear  regression 
models  for  high  and  low  schedule  growth  programs.  As  with  the  cost  growth  models, 
these  models  are  designed  to  be  used  in  conjunction  with  the  logistic  regression  model 
provided  in  Equations  16.  If  the  High  Schedule  Growth?  model  predicts  that  high 
schedule  growth  is  likely  to  occur,  then  the  High  Schedule  Growth  Linear  Regression 
Model  (Equation  17)  can  be  used  to  predict  the  likely  percent  of  schedule  growth.  If  the 
High  Schedule  Growth?  model  predicts  that  high  schedule  growth  is  not  likely  to  occur, 
then  the  Low  Schedule  Growth  Linear  Regression  Model  (Equation  18)  can  be  used  to 
predict  the  likely  percent  of  schedule  growth. 

The  High  Schedule  Growth  Linear  Regression  Model  is: 
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SGh  =  0.899  +  1.188  +  0.587  ^  (Inf  I  Develop)  (17) 

where  SGh  is  the  dependent  variable  and  is  the  predicted  Schedule  Growth  as  a 
percentage  of  the  original  schedule  in  months  from  program  initiation,  JPL  is  a  binary 
variable  with  a  value  of  “1”  for  systems  developed  by  the  Jet  Propulsion  Laboratory  and 
a  value  of  “0”  otherwise,  and  Int’l  Develop  is  a  binary  variable  with  a  value  of  “1”  for 
systems  developed  by  organizations  belonging  to  countries  other  than  the  United  States 
and  a  value  of  “0”  otherwise.  The  positive  coefficients  on  both  JPL  and  Int’l  Develop 
indicate  that  both  of  these  variables  are  associated  with  higher  Schedule  Growth.  The  R 
for  this  model  is  0.79  and  the  adjusted  R  is  0.73.  Appendix  M  provides  the  complete 
output  provided  by  the  JMP  Software  (JMP®,  2005)  for  this  regression.  The  diagnostics 
for  this  model  are:  two  points  have  a  Cook’s  Distance  over  0.5,  the  studentized  residuals 
have  a  Shapiro-Wilk  p-value  of  0.8971,  and  the  residuals  have  a  Breusch-Pagan  p-value 
of  0.0666.  Removal  of  the  two  influential  data  points  that  have  a  Cook’s  Distance  over 
0.5  results  in  the  removal  of  the  two  points  that  represent  the  Int’l  Develop  variable  (total 
sample  size  is  9  observations).  Removal  of  these  two  points  does  not  affect  the  intercept, 
nor  the  coefficient  or  significance  level  for  the  JPL  variable.  The  diagnostics  for  the 
model  without  the  two  influential  data  points  are:  all  points  have  a  Cook’s  Distance 
below  0.3,  the  studentized  residuals  have  a  Shapiro-Wilk  p-value  of  0.3357,  and  the 
residuals  have  a  Breusch-Pagan  p-value  of  0.4450.  Appendix  N  provides  the  complete 
output  provided  by  the  JMP  Software  (JMP  ,  2005)  for  this  regression  with  the  two 
influential  data  points  removed. 

The  Low  Schedule  Growth  Linear  Regression  Model  is: 

SGl  =  0.252  +  0.243  *{Northrop  Grumman)  -  0.274  *{Johns  Hopkins)  (18) 
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where  SGl  is  the  dependent  variable  and  is  the  predicted  Schedule  Growth  as  a 
percentage  of  the  original  schedule  in  months  from  program  initiation,  Northrop 
Grumman  is  a  binary  variable  with  a  value  of  “1”  for  systems  developed  by  Northrop 
Grumman  (or  its  predecessors)  and  a  value  of  “0”  otherwise,  aad  Johns  Hopkins  is  a 
binary  variable  with  a  value  of  “1”  for  systems  developed  by  Johns  Hopkins  University 
and  a  value  of  “0”  otherwise.  The  positive  coefficient  on  Northrop  Grumman  indicates 
that  this  developer  is  associated  with  higher  Schedule  Growth',  whereas  the  negative 
coefficient  on  Johns  Hopkins  indicates  that  this  developer  is  associated  with  lower 
Schedule  Growth.  The  for  this  model  is  0.27  and  the  adjusted  is  0.23.  These  low 
R  values  indicate  that  most  of  the  variation  for  schedule  growth  for  systems  that 
experience  low  schedule  growth  is  due  to  factors  not  explained  by  the  model.  Appendix 
O  provides  the  complete  output  provided  by  the  JMP  Software  (JMP®,  2005)  for  this 
regression.  The  diagnostics  for  this  model  are:  one  point  has  a  Cook’s  Distance  over  0.5, 
the  studentized  residuals  have  a  Shapiro- Wilk  p-value  of  0.8105,  and  the  residuals  have  a 
Breusch-Pagan  p-value  of  0.2848.  Removal  of  the  influential  data  point  does  not  impact 
the  intercept  nor  the  coefficient  or  significance  level  for  the  Johns  Hopkins  variable.  The 
removal  of  the  influential  data  point  increases  the  coefficient  for  Northrop  Grumman 
from  0.243  to  0.41 1  and  improves  the  significance  level  for  this  variable.  The 
diagnostics  for  the  model  without  the  influential  data  point  are:  all  points  have  a  Cook’s 
Distance  below  0.1,  the  studentized  residuals  have  a  Shapiro-Wilk  p-value  of  0.0782,  and 
the  residuals  have  a  Breusch-Pagan  p-value  of  0.2678.  Appendix  P  provides  the 
complete  output  provided  by  the  JMP  Software  (JMP®,  2005)  for  this  regression  with  the 
influential  data  point  removed. 
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Chapter  Summary 

This  chapter  presented  the  Preliminary  and  Inferential  Analysis  for  eost  growth  of 
DoD  systems  and  for  eost  and  schedule  growth  of  NASA  systems.  The  Preliminary 
Analysis  explored  potential  relationships  between  individual  predictors  and  the  responses 
through  graphical  presentation.  The  Preliminary  Analysis  also  identified  correlations 
between  predictor  variables.  The  Inferential  Analysis  presented  linear  and  logistic 
regression  models  for  predicting  cost  and  schedule  growth,  along  with  the  diagnostics 
used  to  assess  these  models.  The  next  ehapter  provides  further  discussion  on  these 
models  and  the  predictors  they  identified  as  signifieant. 
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V.  Discussion 


Chapter  Overview 

This  chapter  highlights  those  faetors  found  to  be  predictive  of  eost  and  sehedule 
growth.  The  diseussion  begins  with  the  predietors  of  Department  of  Defense  (DoD) 
spaee  system  eost  growth  and  then  turns  to  predietors  of  National  Aeronauties  and  Spaee 
Administration  (NASA)  spaee  systems  eost  and  sehedule  growth.  The  ehapter  eoneludes 
with  a  discussion  of  the  limitations  of  the  study  and  reeommendations  for  future  researeh. 
DoD  Predictor  Discussion 

The  models  for  DoD  eost  growth  provided  in  Table  5  reveal  that  eommunieations 
missions,  ground  equipment,  firm-fixed  priee  contracts,  and  increased  program  manager 
tenure  are  all  predietive  of  lower  eost  growth.  One  possible  explanation  for  redueed  eost 
growth  for  eommunieations  missions  and  ground  equipment  is  the  prevalence  of  these 
teehnologies  in  the  eommereial  seetor.  The  widespread  use  and  availability  of  these 
types  of  teehnologies  in  both  public  and  private  sectors  may  make  these  technologies 
more  mature,  and  thus  less  risky,  than  other  missions  and  eommodity  types.  Ground 
equipment  also  benefits  from  the  ability  to  test  in  an  operational  environment,  a  luxury 
that  most  spaee-based  systems  do  not  have. 


Table  5.  DoD  Cost  Growth  Regression  Equations 


Model  Title 

Model 

Fit 

Exclusions 

Total  Cost 

Growth 

CGt  =  0.715  -  Q.511*{Communications) 

0.29 

Adj.  0.24 

Strategic 
Missiles  and 
Titan  IV 

Per  Unit  Cost 
Growth  Model  1 

CGu  =  0.869  -  Q. 9 A\* {Ground Equip)  - 
i).66\*{FFP) 

R"  0.50 

Adj.  R^  0.44 

None 

Per  Unit  Cost 
Growth  Model  2 

CGu  ==  2.069  -  \  {Ground  Equip)  - 

0.664*(/’M  Tenure) 

R^  0.47 

Adj.  R^  0.41 

None 

Per  Unit  Cost 
Growth  Model  3 

CGu  =  0.945  -  \  {Ground Equip)  - 

Q).666*{FFP) 

R"  0.56 

Adj.  R^  0.49 

Strategic 

Missiles 
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The  study  also  found  firm-fixed  priee  contraets  (contraets  with  a  specified 
payment  amount)  to  be  predictive  of  lower  cost  growth.  This  finding  is  consistent  with 
Rossetti’s  (2004)  finding  that  firm-fixed  price  contracts  are  predictive  of  reduced  support 
cost  growth  for  DoD  weapon  systems.  However,  it  is  important  to  remember  that 
regression  analysis  identifies  relationships  but  does  not  indicate  cause  and  effect.  It  could 
be  that  firm-fixed  price  contracts  provide  contractors  with  an  incentive  to  minimize  cost 
growth,  since  additional  costs  reduce  their  profit  margin.  An  alternative  explanation  is 
that  government  programs  use  firm-fixed  price  contracts  on  programs  that  are  relatively 
well  defined,  have  mature  technologies,  and  are  less  risky.  Thus,  while  the  models 
indicate  that  firm-fixed  price  contracts  are  associated  with  reduced  cost  growth,  the 
models  do  not  reveal  whether  these  types  of  contracts  lead  to  lower  cost  growth  or  are 
deliberately  chosen  for  the  types  of  programs  that  would  be  expected  to  have  lower  cost 
growth. 

Both  the  Young  Task  Force  (Defense  Science,  2003)  and  the  Government 
Accountability  Office  (GAO)  (Government,  2006)  studies  identify  high  turnover  of 
Program  Managers  is  a  factor  that  contributes  to  the  cost  growth  of  space  systems.  This 
study  supports  this  assessment,  finding  that  longer  Program  Manager  tenures  are 
predictive  of  lower  cost  growth  (and  thus,  shorter  tenures  are  predictive  of  higher  cost 
growth).  Additionally,  this  study  quantifies  the  impact  of  Program  Manager  tenure, 
finding  that  a  one  year  increase  in  Program  Manager  tenure  is  associated  with  a  reduction 
in  per  unit  cost  growth  of  66.4  percentage  points. 
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Although  a  number  of  DoD  weapons  systems  eost  growth  studies  have  found 


smaller  programs  to  be  assoeiated  with  higher  eost  growth,  this  study  did  not  find 
program  size  to  be  predictive  of  cost  growth  for  space  systems. 

NASA  Predictor  Discussion 
Cost  Growth 

Due  to  the  bimodal  nature  of  the  cost  growth  data  for  the  NASA  dataset,  the 
inferential  analysis  began  with  dividing  the  dataset  into  high  cost  growth  and  low  cost 
growth  programs,  and  then  used  logistic  regression  to  assess  whether  a  program  was 
likely  to  experience  high  or  low  cost  growth  (low  cost  growth  includes  no  cost  growth  as 
well  as  negative  cost  growth).  Table  6  provides  the  NASA  cost  growth  models. 


Table  6.  NASA  Cost  Growth  Regression  Equations 


Model 

Title 

Model 

Fit 

High  Cost 
Growth? 
Logistic 
Model  1 

T  ^  2.140  -  0.05^* {Initial Program  Size)  +  Qm\*{Total Mass) 

i-HCG  ”  e 

j  ^  2.140  -0.0S^*{lmtial Program  Size) +  ^m\*{Total Mass) 

(U)  0.57 

High  Cost 
Growth? 
Logistic 
Model  2 

T  _  0.741  -0.0'I)^*{Initial Program  Size)  +  0.00\*{Dry  Mass)  +  7>%.105*{Microgravity) 

i-HCG  “  e 

2  _l_  ^  0.741  -Q.0'i%*{lmtial  Program  Size)  +  i)X)0\*{Dry  Mass)  +  38.705*{Micragrav/(y) 

(U)  0.50 

High  Cost 

Growth 

Linear 

Model 

CGh  =  1.232  +  \.t)'il*{Space  Physics) 

0.77 

Adj.  0.74 

Low  Cost 

Growth 

Linear 

Model 

CGl  =  0.509  -  A* (Program  Start) 

R^O.18 

Adj.  0.16 
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From  the  logistic  regression  analysis,  this  study  found  that  larger  program  size 
(measured  in  total  cost)  decreased  the  likelihood  of  being  a  high  cost  growth  program, 
whereas  more  massive  spacecrafts  and  microgravity  missions  increased  the  likelihood  of 
being  a  high  cost  growth  program.  This  finding  of  larger  programs  being  associated  with 
lower  cost  growth  is  consistent  with  many  other  cost  growth  studies  (Schaffer,  2004; 
McCrillis,  2003;  Dameron  et  al.,  2002;  Pannell,  1994:42;  Drezner  et  al.,  1993:27).  As 
discussed  in  Chapter  II,  smaller  programs  are  more  likely  to  experience  high  cost  growth 
due  to  minimal  oversight  and  because  equivalent  costs  and  increases  in  costs  represent 
proportionally  greater  amounts  of  the  total  cost  for  smaller  programs  (Drezner  et  ah, 
1993:49).  Further  study  is  recommend  to  determine  the  cause  of  the  increased  likelihood 
of  high  cost  growth  for  more  massive  spacecraft  and  microgravity  missions.  While  this 
increased  likelihood  could  be  an  indication  of  the  increased  technical  complexity  of  these 
types  of  systems,  it  may  also  be  an  indication  of  other  problems  unique  to  these  programs 
such  as  inadequate  cost  estimating  procedures,  deficient  program  acquisition  processes, 
or  other  technical  or  scientific  issues. 

After  using  the  logistic  regression  to  determine  the  likelihood  of  high  cost  growth, 
the  linear  regression  models  are  then  used  for  determining  the  quantity  of  cost  growth. 

For  those  programs  that  are  likely  to  experience  high  cost  growth,  the  amount  of  cost 
growth  increases  for  those  programs  from  a  space  physics  mission.  Again,  further  study 
is  recommended  to  identify  the  root  causes  for  this  relationship. 

For  programs  in  which  the  logistic  models  predict  to  be  likely  to  experience  low 
cost  growth,  program  start  date  is  the  best  predictor  of  the  amount  of  cost  growth,  with 
more  recent  programs  associated  with  lower  cost  growth.  Further  study  is  recommended 
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to  determine  if  this  relationship  is  an  indieation  of  improved  program  aequisition  or  eost 
estimating  proeesses. 

Schedule  Growth 

Similar  to  the  eost  growth  analysis  for  NASA  spaee  systems,  the  sehedule  growth 
dataset  also  displays  a  bimodal  distribution.  Thus,  the  inferential  analysis  began  with 
dividing  the  dataset  into  high  sehedule  growth  and  low  sehedule  growth  programs,  and 
then  used  logistie  regression  to  assess  whether  a  program  was  likely  to  have  high  or  low 
sehedule  growth.  After  determining  whether  or  not  high  sehedule  growth  was  likely,  the 
linear  regression  models  are  then  used  to  determine  the  amount  of  likely  growth.  Table  7 
provides  the  NASA  sehedule  growth  models. 


Table  7.  NASA  Sehedule  Growth  Regression  Equations 


Model  Title 

Model 

Fit 

High 

Schedule 

Growth? 

Logistic 

Model 

^  +  0.00\*(Final Program  Size) 

^  -2.186  Program  Size) 

R^(U)  0.15 

High 

Schedule 

Growth 

Linear  Model 

SGh  =  0.899  +  1.188  *{JPL)  +  0.587  Develop) 

0.79 

Adj.  R^  0.73 

Low 

Schedule 

Growth 

Linear  Model 

SGl  =  0.252  +  0.243  ^{Northrop  Grumman)  -  0.274  *{Johns 
Hopkins) 

R^  0.27 

Adj.  R^  0.23 

The  logistie  regression  results  found  that  larger  programs  (measured  in  total  eost) 
are  more  likely  to  experienee  high  sehedule  growth.  For  those  programs  likely  to 
experienee  high  sehedule  growth,  the  linear  regressions  reveal  that  those  programs 
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developed  by  JPL  or  an  International  developer  (outside  of  the  U.S.)  experienee  a  greater 
amount  of  sehedule  growth.  For  those  programs  likely  to  experience  low  schedule 
growth,  those  developed  by  Northrop  Grumman  are  associated  with  increased  schedule 
growth,  whereas  those  space  systems  developed  by  Johns  Hopkins  are  associated  with  a 
reduced  amount  of  schedule  growth.  Keep  in  mind  that  these  results  do  not  indicate 
cause  and  effect;  more  research  is  needed  to  discover  whether  these  developers  have 
processes  that  actually  lead  to  schedule  growth  (or  reduced  growth  in  the  case  of  Johns 
Hopkins)  or  if  they  are  more  likely  to  take-on  complex  projects  that  possess  other  factors 
leading  to  schedule  growth. 

Interestingly,  the  study  did  not  find  a  predictive  relationship  between  cost  and 
schedule  growth  for  NASA  space  systems;  many  of  the  programs  that  experienced  cost 
growth  did  not  experience  schedule  growth  and  vice  versa.  Note  that  the  study  did  not 
find  these  variables  to  be  negatively  correlated  either;  that  is,  it  does  not  appear  that  the 
programs  avoid  one  type  of  growth  by  permitting  the  other  (e.g.,  increasing  costs  in  order 
to  reduce  schedule  slip).  The  finding  that  these  types  of  growth  are  not  strongly 
correlated,  and  thus  have  different  factors  influencing  each,  is  consistent  with  the  findings 
of  Foreman  (2007)  and  Drezner  and  Smith  (1990). 

Conclusions 

This  study  provides  defense  and  civil  cost  estimators  and  space  system  acquirers 
with  a  set  of  models  to  aid  in  predicting  cost  and  schedule  growth.  Since  many  of  the 
systems  that  the  defense  space  acquisition  community  will  be  tasked  to  acquire  will  be 
systems  other  than  ground  equipment  and  communications  systems,  the  analysis  suggests 
that  cost  estimators  and  acquirers  should  anticipate  that  other  systems  are  likely  to 
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experience  higher  cost  growth,  and  should  plan  accordingly.  Additionally,  this  research 
indicates  that  longer  Program  Manager  tenures  are  associated  with  decreased  cost  growth. 
The  respective  model  predicts  that  increasing  the  average  Program  Manager  tenure  by 
one-year  will  reduce  the  anticipated  per  unit  cost  growth  by  66.4  percentage  points. 

Thus,  this  research  supports  the  recommendation  of  Young’s  Task  Force  to  increase  the 
length  of  Program  Managers’  tenures  (Defense  Science,  2003). 

Similarly,  while  NASA  will  continue  to  procure  a  variety  of  systems,  with  wide 
ranges  of  program  sizes,  spacecraft  sizes,  and  mission  types,  it  would  behoove  cost 
estimators  and  acquirers  to  recognize  that  smaller  programs,  more  massive  spacecraft, 
and  microgravity  and  space  physics  missions  are  more  vulnerable  to  experiencing  higher 
cost  growth.  Additionally,  cost  estimators  and  acquirers  should  also  recognize  that  while 
larger  programs  are  less  vulnerable  to  cost  growth,  they  are  more  vulnerable  to  schedule 
growth. 

Study  Limitations 

This  study  is  an  exploratory  analysis,  intended  to  provide  a  starting  point  for 
developing  space  system  cost  and  schedule  growth  models  for  use  by  space  system 
acquirers  and  cost  estimators.  The  study  seeks  not  only  to  identify  the  best  predictors  of 
cost  and  schedule  growth,  but  also  to  identify  an  appropriate  methodology  for  acquirers 
and  cost  estimators  to  use  in  establishing  their  own  models.  While  linear  regression 
analysis  is  sufficient  for  DoD  space  programs,  it  is  not  suitable  on  its  own  for  NASA 
space  systems.  Instead,  the  analysis  found  that  NASA  systems  have  bimodal 
distributions  that  are  best  modeled  by  first  using  logistic  regression  to  determine  if  a 
program  was  likely  to  experience  high  or  low  growth,  and  then  using  linear  regression 
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models  to  predict  the  likely  amount  of  growth.  While  most  of  the  models  are  highly 
predictive,  users  of  the  models  should  keep  these  limitations  in  mind: 


•  This  study  is  a  starting  point  for  quantitatively  assessing  space  system  cost  growth 
using  regression  analysis.  The  models  should  be  further  validated  with  space 
systems  outside  of  the  original  dataset.  Until  this  has  been  accomplished,  the 
models  are  only  known  to  be  predictive  of  programs  within  the  dataset. 

•  The  study  identified  a  bimodal  relationship  for  cost  and  schedule  growth  among 
NASA  programs,  however  the  number  of  programs  belonging  to  the  high  cost 
growth  and  high  schedule  growth  categories  is  relatively  small.  The  NASA 
dataset  should  be  augmented  with  additional  programs  to  verify  this  bimodal 
relationship. 

•  This  study  identifies  and  quantifies  those  factors  that  are  best  predictors  of  space 
system  cost  and  schedule  growth;  however,  it  does  not  include  the  depth  of  a 
qualitative  study  in  investigating  the  root  causes  of  these  relationships. 

•  This  study  included  only  DoD  Major  Defense  Acquisition  Programs  (MDAPs) 
reported  in  the  Selected  Acquisition  Report  and  select  NASA  spacecraft.  The 
study  does  not  account  for  commercial  space  systems,  classified  space  systems, 
NRO  space  systems,  or  non-MDAP  programs. 

•  There  are  many  potential  predictors  of  cost  and  schedule  growth  that  were  not 
included  in  the  analysis  due  to  the  lack  of  available  data.  Potential  predictors  not 
evaluated  include:  schedule  milestones,  number  of  requirements  and  requirements 
changes,  amount  of  systems  engineering  expertise,  and  level  of  technology 
maturity. 


105 


•  The  models  can  only  be  as  sound  as  the  data  from  which  they  are  derived. 
Inconsistencies  in  data  format  such  as  which  costs  are  included  in  the  total  figure, 
the  level  of  technology  maturity  when  the  program  was  initiated,  and  the 
combination  or  division  of  multiple  satellites  into  a  single  or  several  programs,  all 
affect  the  accuracy  of  the  models. 

•  The  study  also  includes  some  assumptions  for  the  inflation  and  quantity 
adjustments.  For  example,  when  adjusting  DoD  Total  Cost  Growth  for  quantity, 
the  current  estimate  was  adjusted  by  omitting  the  Quantity  Variance  category 
from  the  SAR.  This  assumes  that  all  cost  growth  related  to  quantity  is  captured  in 
this  category.  With  the  NASA  datasets,  the  final  cost  was  adjusted  for  inflation 
by  assuming  that  all  costs  were  in  then  year  dollars  for  the  launch  year  rather  than 
breaking  the  costs  down  by  each  year  they  were  incurred  and  adjusting  each  year 
for  inflation  separately.  It  was  assumed  that  the  bulk  of  funds  are  spent  at  the  tail 
end  of  the  program  and  that  assuming  all  costs  were  incurred  during  the  launch 
year  would  be  a  close  approximation.  While  the  inflation  and  quantity 
adjustments  are  not  perfect,  they  better  capture  reality  than  if  no  adjustments  were 
made  at  all. 

Recommendations  for  Future  Research 

From  this  study  stems  numerous  avenues  for  further  research.  Potential  future 
areas  for  study  include; 

•  Test  models  provided  herein  with  additional  data  from  other  NASA  and  DoD 
programs  to  validate  models  or  establish  more  robust  models. 
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•  Test  models  provided  herein  using  data  from  commereial  and  elassified  systems 
to  see  if  the  models  apply  or  if  new  models  are  needed, 

•  Explore  additional  predietor  variables  not  evaluated  herein,  such  as  requirements, 
systems  engineering  expertise,  or  technological  maturity, 

•  Augment  NASA  data  with  additional  space  programs  to  see  if  the  bimodal 
distributions  for  cost  and  schedule  growth  hold,  or 

•  Further  explore  relationships  identified  herein  using  a  more  in-depth  qualitative 
analysis. 

Chapter  Summary 

This  study  provides  a  foundation  for  predicting  space  system  cost  and  schedule 
growth.  It  explored  numerous  programmatic  characteristics  to  identify  those  that  are  best 
predictors  of  growth.  The  study  provides  four  models  for  use  in  predicting  DoD  space 
system  cost  growth.  It  also  identified  a  bimodal  distribution  for  cost  and  schedule  growth 
of  NASA  space  systems,  and  thus  established  a  series  of  logistic  and  linear  models  to 
assist  in  cost  and  schedule  growth  forecasting  for  NASA  space  systems. 
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Appendix  A,  List  of  DoD  Space  Systems 


System  Name 

Initial 

Selected 

Acquisition 

Report 

(SAR) 

Description 

Advanced  Extremely  High  Frequeney 
(AEHF) 

2002 

Satellite 

Evolved  Expendable  Eauneh  Vehiele 
(EEEV) 

1999 

Eauneh  Vehiele 

Global  Broadcast  Service  (GBS) 

1997 

Ground  Equipment 

Minuteman  111  Guidanee  Replaeement 
Program  (GRP) 

1993 

Strategie  Missile  Upgrade 

Minuteman  111  Propulsion  Replaeement 
Program  (PRP) 

1996 

Strategie  Missile  Upgrade 

NAVSTAR  Global  Positioning  System 
(GPS) 

1980 

Satellite 

NAVSTAR  Global  Positioning  System 
(GPS)  Equipment 

1980 

Ground  Equipment 

National  Polar-orbiting  Operational 
Environmental  Satellite  System 
(NPOESS) 

2002 

Satellite 

Spaee-Based  Infrared  System  (SBIRS) 
High 

1997 

Satellite 

Wideband  Global  SATCOM  (WGS) 

2001 

Satellite 

Mobile  User  Objeetive  System 
(MUOS) 

2004 

Satellite 

Defense  Satellite  Communieation 

System  (DSCS)  111 

1977 

Satellite 

Defense  Support  Program  (DSP) 

1983 

Satellite 

Inertial  Upper  Stage  (lUS) 

1982 

Eauneh  Vehiele  Upper  Stage 

Peaeekeeper 

1983 

Strategie  Missile 

Titan  IV 

1985 

Eauneh  Vehiele 

Minuteman  11 

1969 

Strategie  Missile 

Minuteman  111 

1969 

Strategie  Missile 

Defense  Meteorologieal  Satellite 
Program  (DMSP) 

1983 

Satellite 

MIESTAR 

1992 

Satellite 

Single  Channel  Anti-jam  Man-Portable 
Terminal  (SCAMP) 

1992 

Ground  Equipment 
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Appendix  B,  List  of  NASA  Space  Systems 


Program  Name 

Initial  budget 
year 

Launch 

Date 

Full  Name 

ACE 

1994 

Aug-97 

Advanced  Composition  Explorer 
Satellite 

ACTS 

1983 

Sep-93 

Advanced  Communications 
Technology  Satellite 

AE-C 

1971 

Dee-73 

Atmosphere  Explorer-C 

AEM-HCMM 

1974 

Apr-78 

Application  Explorer  Mission-Heat 
Capacity  Mapping  Mission 

ATS-1 

Not  Available 

Dee-66 

Applications  Technology  Satellite  1 

ATS-2 

Not  Available 

Apr-67 

Applications  Technology  Satellite  2 

ATS-5 

Not  Available 

Aug-69 

Applications  Technology  Satellite  5 

ATS-6 

1968 

May-74 

Applications  Technology  Satellite  6 

Aura 

1993 

Jul-04 

Aura 

AXAE 

1990 

Jul-99 

(Chandra)  Advanced  X-Ray 
Astrophysics  Pacility 

CALIPSO 

1999 

Apr-06 

Cloud- Aerosol  Eidar  and  Infrared 
Pathfinder  Satellite  Observations 

Cassini 

1990 

Oet-97 

Cassini 

CloudSat 

1999 

Apr-06 

CloudSat 

COBE 

1982 

Nov-89 

Cosmic  Background  Explorer 

CONTOUR 

2000 

Jul-02 

Comet  Nucleus  Tour 

COSTR 

1987 

Jul-92 

Collaborative  Solar  Terrestrial 
Research 

Deep  Spaee  1 

1996 

Oot-98 

Deep  Space  1 

DSCS-2 

1969 

Nov-71 

Defense  Satellite  Communications 
System 

Endeavour 

1987 

May-92 

Shuttle  Orb  iter  Endeavour  (OV- 
105) 

EO-1 

1996 

Nov-00 

Earth  Observing  One 

ESSP  (VCE/GRACE) 

1997 

Mar-02 

Earth  Systems  Science  Pathfinder 
(Vegetation  Canopy  Eidar/Gravity 
Recovery  and  Climate  Experiment) 

EUVE 

1984 

Jun-92 

Extreme  Ultraviolet  Explorer 

PAST 

1989 

Aug-96 

Past  Auroral  Snapshot  Explorer 

PUSE 

1995 

Jun-99 

Par  Ultraviolet  Spectroscopic 
Explorer 

GAEEX 

1998 

Apr-03 

Galaxy  Evolution  Explorer 

Galileo 

1978 

Oot-89 

Galileo 

Genesis 

1998 

Aug-01 

Genesis 

GOES  1-M 

1984 

Apr-94 

Geostationary  Operational 
Environmental  Satellite 
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GRO 

1981 

Apr-91 

(Compton)  Gamma  Ray 

Observatory 

HEAO-A 

1972 

Aug-77 

High  Energy  Astronomical 
Observatory 

HESSI 

1998 

Eeb-02 

High  Energy  Solar  Spectroscopic 
Imager  (now  RHESSI) 

HETE-II 

1997 

Oct-00 

High  Energy  Transient 
Experiment/Explorer 

HST 

1977 

Apr-90 

Hubble  Space  Telescope 

ICESat 

1996 

Jan-03 

Ice,  Clouds  and  Eand  Elevation 
Satellite 

IMAGE 

1996 

Mar-00 

Imager  for  Magnetopause  to  Aurora 
Global  Exploration 

EANDSAT-A 

1969 

Jul-72 

Land  Remote  Sensing  Satellite 

EANDSAT-D 

1977 

Jul-82 

Land  Remote  Sensing  Satellite 

Eunar  Orbiter 

1964 

Aug-66 

Lunar  Orbiter 

Eunar  Prospector 

1996 

Jan-98 

Lunar  Prospector  Orbiter 

Magellan 

1984 

May-89 

Magellan 

MAP 

1996 

Jun-01 

(Wilkinson)  Microwave  Anisotropy 
Probe 

Mars  Observer 

1985 

Sep-92 

Mars  Observer 

Mars  Odyssey 

1998 

Apr-01 

Mars  Odyssey 

Mars  Pathfinder 

1994 

Dec-96 

Mars  Pathfinder 

MCO 

1996 

Dec-98 

Mars  Climate  Orbiter 

MER 

2000 

Jun-03 

Mars  Exploration  Rover 

MGS 

1994 

Nov-96 

Mars  Global  Surveyor 

NEAR 

1994 

Eeb-96 

Near  Earth  Asteroid  Rendezvous 
(Shoemaker) 

NSCAT 

1985 

Aug-96 

NASA  Scatterometer 

OSO-8 

1970 

Jun-75 

Orbiting  Solar  Observatory 

SCATHA 

1976 

Jan-79 

Spacecraft  Charging  at  High 
Altitudes 

SIRTE 

1996 

Aug-03 

(Spitzer)  Space  Infrared  Telescope 
Eacility 

Skylab  Workshop 

1969 

May-73 

Skylab  Workshop 

SMS-1 

1970 

May-74 

Synchronized  Meteorology  Satellite 

SORCE 

1999 

Jan-03 

Solar  Radiation  and  Climate 
Experiment 

Space  Station 

1987 

Nov-98 

International  Space  Station  Alpha 
(ISSA) 

Spacelab 

1974 

Nov-83 

Spacelab 

STARDUST 

1996 

Eeb-99 

Star  Dust 

no 


SWAS/TRACE/WIRE 

1989 

Apr-98 

Submillimeter  Wave  Astronomy 
Satellite/Transition  Region  and 
Coronal  Explorer/Wide -Eield 
Infrared  Explorer 

TDRSS  replen 

1994 

Jun-00 

Tracking  and  Data  Relay  Satellite  - 
3  replenishment  satellites 

TDRSS-7 

1986 

Jul-95 

Tracking  and  Data  Relay  Satellite 

Terra 

1991 

Dec-99 

EOS  AM-1 

TIMED 

1997 

Dec-01 

Thermosphere,  Ionosphere, 
Mesosphere  Energetics  and 
Dynamics 

TIROS-M 

Not  Available 

Jan-70 

Television  Infrared  Observation 
Satellite  (also  lTOS-1) 

TOPEX 

1987 

Aug-92 

Ocean  Topography 
Experiment/Poseidon 

TRMM 

1991 

Nov-97 

Tropical  Rainfall  Measuring 

Mission 

TSS 

1984 

Jul-92 

Tethered  Satellite  System 

UARS 

1982 

Sep-91 

Upper  Atmosphere  Research 

Satellite 

Ulysses 

1979 

Oct-90 

Ulysses 

Viking  Eander 

1970 

Aug-75 

Viking  Eander 

XTE 

1990 

Dec-95 

(Rossi)  X-Ray  Timing  Explorer 
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Appendix  C,  DoD  Total  Cost  Growth  Linear  Regression  Model  Output 


Whole  Model 
Regression  Plot 


Msn:  Communications 
Leverage  Plot 


Actual  by  Predicted  Plot 


Residual  by  Predicted  Plot 


Summary  of  Fit 


RSquare  0.290373 

RSquare  Adj  0.235787 

Root  Mean  Square  Error  0.483612 

Mean  of  Response  0.445538 

Observations  (or  Sum  Wgts)  15 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

1 

1.2441269 

1.24413 

5.3195 

Error 

13 

3.0404483 

0.23388 

Prob  >  F 

C.  Total 

14 

4.2845752 

0.0382 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Std  Beta 

Min  Value  Max  Value 

Intercept 

0.7149335 

0.170983 

4.18 

0.0011 

0 

Msn:  Communications 

-0.577277 

0.250293 

-2.31 

0.0382 

-0.53886 

0  1 

112 


Appendix  D,  DoD  Per  Unit  Cost  Growth  Linear  Regression  Model  1  Output 


Ground  Equip  FFP 

Leverage  Plot  Leverage  Plot 


Whole  Model 

Actual  by  Predicted  Plot 


Residual  by  Predicted  Plot 


o 

O 


1.0- 


0.5- 


dj 

a: 

£  0.0- 


-0.5- 


-1.0- 


— 

-1.0  -0.5 


"T” 

.0 


T 


T 


T" 


.5  1.0  1.5 

%  Per  Unit  Cost 
Growth  Predicted 


2.0 


Summary  of  Fit 


RSquare  0.496146 

RSquareAdj  0.440163 

Root  Mean  Square  Error  0.508769 

Mean  of  Response  0.424743 

Observations  (or  Sum  Wgts)  21 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Modei 

2 

4.5879632 

2.29398 

8.8623 

Error 

18 

4.6592316 

0.25885 

Prob  >  F 

C.  Totai 

20 

9.2471948 

0.0021 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio  Prob>|t| 

Std  Beta 

Min  Vaiue 

Max  Vaiue 

intercept 

0.8693884 

0.155493 

5.59  <.0001 

0 

Ground  Equip 

-0.941423 

0.263987 

-3.57  0.0022 

-0.60425 

0 

1 

FFP 

-0.661491 

0.238515 

-2.77  0.0125 

-0.46992 

0 

1 
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Appendix  E,  DoD  Per  Unit  Cost  Growth  Linear  Regression  Model  1  Output 

Excludes  GPS  User  Equipment 


Whole  Model 

Actual  by  Predicted  Plot 


Residual  by  Predicted  Plot 


1.0- 


O 

O 


CD 

Q. 


CD 

cr 


$ 

p 


0.5- 

0.0-- 


O 


-0.5- 


Ji - ^ - 1 - ^ - 1 - 1 

-0.5  .0  .5  1.0  1.5  2.0 

%  Per  Unit  Cost 
Growth  Predicted 


Ground  Equip 
Leverage  Plot 


FFP 

Leverage  Plot 


Summary  of  Fit 


RSquare  0.382844 

RSquareAdj  0.310237 

Root  Mean  Square  Error  0.5221 14 

Mean  of  Response  0.489074 

Observations  (or  Sum  Wgts)  20 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

2 

2.8747826 

1 .43739 

5.2728 

Error 

17 

4.6342448 

0.27260 

Prob  >  F 

C.  Total 

19 

7.5090275 

0.0165 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Std  Beta 

Min  Value 

Max  Value 

Intercept 

0.856553 

0.165107 

5.19 

<.0001 

0 

Ground  Equip 

-0.896499 

0.308887 

-2.90 

0.0099 

-0.58524 

0 

1 

FFP 

-0.627263 

0.269618 

-2.33 

0.0326 

-0.46912 

0 

1 
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Appendix  F,  DoD  Per  Unit  Cost  Growth  Linear  Regression  Model  2  Output 


Whole  Model 

Actual  by  Predicted  Plot  Residual  by  Predicted  Plot 


Ground  Equip  PM  Tenure 

Leverage  Plot  Leverage  Plot 


Summary  of  Fit 


RSquare  0.467177 

RSquare  Adj  0.407974 

Root  Mean  Square  Error  0.523191 

Mean  of  Response  0.424743 

Observations  (or  Sum  Wgts)  21 


Analysis  of  Variance 


Source 

DF  Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

2 

4.3200722 

2.16004 

7.8911 

Error 

18 

4.9271226 

0.27373 

Prob  >  F 

C.  Total 

20 

9.2471948 

0.0035 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio  Prob>|t| 

Std  Beta 

Min  Value 

Max  Value 

Intercept 

2.0692045 

0.591726 

3.50  0.0026 

0 

Ground  Equip 

-1.177888 

0.302594 

-3.89  0.0011 

-0.75602 

0 

1 

PM  Tenure 

-0.664477 

0.264845 

-2.51  0.0219 

-0.48728 

1 

3 
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Appendix  G,  DoD  Per  Unit  Cost  Growth  Linear  Regression  Model  3  Output; 

Excludes  Strategic  Missiles 


Whole  Model 

Actual  by  Predicted  Plot 


Ground  Equip 
Leverage  Plot 


Residual  by  Predicted  Plot 

1.0-1 - 


^  _  0.5- 

CO  CO 

o  ^ 

o  :s 

I  I  00- 


o  -0.5- 
U 


-1.0- 


-1.0  -0.5  .0  .5  1.0  1.5  2.0 

%  Per  Unit  Cost 
Growth  Predicted 


FFP 

Leverage  Plot 


Summary  of  Fit 


RSquare  0.561866 

RSquare  Adj  0.49446 

Root  Mean  Square  Error  0.531616 

Mean  of  Response  0.479506 

Observations  (or  Sum  Wgts)  16 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

2 

4.7115613 

2.35578 

8.3356 

Error 

13 

3.6740053 

0.28262 

Prob  >  F 

C.  Total 

15 

8.3855666 

0.0047 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Std  Beta  Min  Value 

Max  Value 

Intercept 

0.9453955 

0.181404 

5.21 

0.0002 

0 

Ground  Equip 

-1.153119 

0.340798 

-3.38 

0.0049 

-0.6217 

0 

1 

FFP 

-0.665813 

0.27476 

-2.42 

0.0307 

-0.44525 

0 

1 
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Appendix  H,  NASA  Cost  Growth  Logistic  Regression  Model  1  Output 


Whole  Model  Test 

Model  -LogLikelihood  DF 

Difference  10.683222  2 

Full  8.153711 

Reduced  18.836933 

RSquare  (U)  0.5671 

Observations  (or  Sum  Wgts)  54 


Converged  by  Gradient 


ChiSquare 

21.36644 


Lack  Of  Fit 


Source 

DF 

-LogLikelihood 

ChiSquare 

Lack  Cf  Fit 

51 

8.1537110 

16.30742 

Saturated 

53 

0.0000000 

Prob>ChiSq 

Fitted 

2 

8.1537110 

1.0000 

Parameter  Estimates 

Term 

Intercept 

Initial  Program  Size  (Original 
Estimate  CY07) 

Total  Mass  (kg) 

For  log  odds  of  0/1 


Estimate  Std  Error  ChiSquare 

-2.139853  1.5549805  1.89 

0.05798487  0.0265758  4.76 

-0.001285  0.0006474  3.94 


Effect  Wald  Tests 

Source 

Initial  Program  Size  (Original  Estimate  CY07) 
Total  Mass  (kg) 


Nparm  DF 

1  1 

1  1 


Effect  Likelihood  Ratio  Tests 

Source  Nparm  DF 

Initial  Program  Size  (Original  Estimate  CY07)  1  1 

Total  Mass  (kg)  1  1 


Receiver  Operating  Characteristic 

1.00 

0.90 


0.80 
0.70 

(D 

.^0.60 
'C/D  .> 

(£  S  0.50 

<D  0 

2  OT  0.40 

I- 

0.30 
0.20 
0.10 
0.00 

.00  .10  .20  .30  .40  .50  .60  .70  .80  .901.00 
1  -Specificity 
False  Positive 


Using  High  Cost  Growth?='T  to  be  the  positive  level 
Area  Under  Curve  =  0.95139 


H — 1 — 

— — 1 

— — 1 

— — 

— — 

Prob>ChiSq 

<.0001 


Prob>ChiSq  Mm  Value  Max  Value 


0.1688 

0.0291 

0.0472 


Wald  ChiSquare 
4.7605558 
3.93933688 


L-R  ChiSquare 
21.3478893 
14.0306169 


9.9 

27802 

124 

109000 

Prob>ChiSq 

0.0291 

0.0472 


Prob>ChiSq 

<.0001 

0.0002 
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Appendix  I,  NASA  Cost  Growth  Logistic  Regression  Model  2  Output 


Whoie  Modei  Test 

Modei  -LogLikeiihood 

DF 

ChiSquare 

Prob>ChiSq 

Difference  13.238856 

Fuii  13.049413 

Reduced  26.288270 

3 

26.47771 

<.0001 

RSquare  (U) 

0.5036 

Observations  (or  Sum  Wgts) 

66 

Converged  by  Gradient 


Lack  Of  Fit 


Source 

DF 

-LogLikeiihood 

ChiSquare 

Lack  Of  Fit 

62 

13.049413 

26.09883 

Saturated 

65 

0.000000 

Prob>ChiSq 

Fitted 

3 

13.049413 

1.0000 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

ChiSquare 

Prob>ChiSq 

Min  Vaiue  Max  Vaiue 

intercept 

-0.7413099 

1.0158722 

0.53 

0.4656 

initiai  Program  Size  (Originai 

Estimate  CY07) 

0.03752431 

0.0166158 

5.10 

0.0239 

9.9 

27802 

Dry  Mass  (kg) 

-0.0008909 

0.0003625 

6.04 

0.0140 

117 

90607 

Msn:  Microgravity 

For  iog  odds  of  0/1 

-38.70531 1 

83316.394 

0.00 

0.9996 

0 

1 

Effect  Waid  Tests 

Source 

Nparm 

DF  Waid  ChiSquare 

Prob>ChiSq 

initiai  Program  Size  (Originai  Estimate  CY07) 

1 

1 

5.10013649 

0.0239 

Dry  Mass  (kg) 

1 

1 

6.03971535 

0.0140 

Msn:  Microgravity 

1 

1 

2.15814e-7 

0.9996 

Effect  Likelihood  Ratio  Tests 

Source 

Nparm 

DF 

L-R  ChiSquare 

Prob>ChiSq 

initiai  Program  Size  (Originai  Estimate  CY07) 

1 

1 

23.2927602 

<.0001 

Dry  Mass  (kg) 

1 

1 

17.3608073 

<.0001 

Msn:  Microgravity 

1 

1 

9.25815588 

0.0023 

Receiver  Operating  Characteristic 


/ 

DO 

/ 

0 

'co  .> 

J 

t/j  u.ou 

0  0 

1- 

U.OVJ 

.0 

0  .1 

0  .2 

0  .3 

0 

1- 

Fa 

0  .3 

Spe 

Ise 

0  .6 

cifici 

^osi 

0  .7 

ty 

ive 

0  .8 

0  .£ 

01. 

Using  High  Cost  Growth?='1'  to  be  the  positive  ievei 
Area  Under  Curve  =  0.93957 
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Appendix  J,  NASA  High  Cost  Growth  Linear  Regression  Model  Output 


Whole  Model 

Regression  Plot  Actual  by  Predicted  Plot 


Residual  by  Predicted  Plot 


Msn:  Space  Physics 
Leverage  Plot 


Summary  of  Fit 


RSquare 

0.769929 

RSquare  Adj 

0.737061 

Root  Mean  Square  Error 

0.319372 

Mean  of  Response 

1 .69328 

Observations  (or  Sum  Wgts) 

9 

Analysis  of  Variance 

Source  DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model  1 

2.3893554 

2.38936 

23.4253 

Error  7 

0.7139911 

0.10200 

Prob  >  F 

C.  Total  8 

3.1033465 

0.0019 

Parameter  Estimates 

Term 

Estimate  Std  Error 

t  Ratio  Prob>|t| 

Std  Beta 

Min  Value  Max  Value 

Intercept 

1.2324251  0.142828 

8.63  <.0001 

0 

Msn:  Space  Physics 

1.0369233  0.214242 

4.84  0.0019 

0.877456 

0  1 
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Appendix  K,  NASA  Low  Cost  Growth  Linear  Regression  Model  Output 


Whole  Model 

Regression  Plot  Actual  by  Predicted  Plot 


Residual  by  Predicted  Plot 


0.6-r 

0.4- 


ro 


0.2- 

0.0- 


2 
w 
o 
O 


-0.2- 
-0.4- 
-0.6- 
-0.8- 


-1.0 - 

-0.6 


n - r“ 

-0.4  -0.2 


T 

.0 


T 

.2 


"T 

.4 


"T 

.6 


T - 

.8  1.0 


%  Cost  Growth  Predicted 


Program  Start  (Yrs  from  Baseline) 


Summary  of  Fit 


RSquare  0.179217 

RSquareAdj  0.16456 

Root  Mean  Square  Error  0.290302 

Mean  of  Response  0.187642 

Observations  (or  Sum  Wgts)  58 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

1 

1.0304816 

1 .03048 

12.2276 

Error 

56 

4.7194180 

0.08428 

Prob  >  F 

C.  Total 

57 

5.7498996 

0.0009 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Std  Beta 

Min  Value  Max  Value 

Intercept 

0.50853 

0.099368 

5.12 

<.0001 

0 

Program  Start  (Yrs  from  Baseline) 

-0.013585 

0.003885 

-3.50 

0.0009 

-0.42334 

0  36 
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Appendix  L,  NASA  Schedule  Growth  Logistic  Regression  Model  Output 


Receiver  Operating  Characteristic 


DO 

CD 

‘co  > 

c 

0)  CD 

H 

U.OU 

— * 

.0 

0  .1 

0  .2 

0  .3 

0 

1- 

Fa 

0  .E 

Spe 

Ise 

0  .6 

cific 

^osit 

0  .7 

ty 

ive 

0  .8 

0  .£ 

01. 

Using  High  Scheduie  Growth?='1'  to  be  the  positive  ievei 
Area  Under  Curve  =  0.76010 


Whole  Model  Test 


Modei 

Difference 

Fuii 

Reduced 


-LogLikeiihood 

3.747015 

21.826393 

25.573407 


DF  ChiSquare  Prob>ChiSq 

1  7.49403  0.0062 


RSquare  (U)  0.1465 

Observations  (or  Sum  Wgts)  47 


Converged  by  Gradient 


Lack  Of  Fit 


Source 

DF 

-LogLikeiihood 

ChiSquare 

Lack  Of  Fit 

44 

21.826393 

43.65279 

Saturated 

45 

0.000000 

Prob>ChiSq 

Fitted 

1 

21.826393 

0.4864 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

ChiSquare 

Prob>ChiS 

Min  Vaiue 

intercept 

Finai  Program  Size  (Actuai  Cost  CY07) 

2.18565739 

-0.0013652 

0.5739752 

0.0005352 

14.50 

6.51 

P 

0.0001 

0.0107 

23 

Max  Vaiue 


38589 


For  iog  odds  of  0/1 


Effect  Wald  Tests 

Source 

Finai  Program  Size  (Actuai  Cost  CY07) 


Nparm  DF 

1  1 


Waid  ChiSquare  Prob>ChiSq 

6.50744631  0.0107 


Effect  Likelihood  Ratio  Tests 

Source 

Finai  Program  Size  (Actuai  Cost  CY07) 


Nparm  DF 

1  1 


L-R  ChiSquare  Prob>ChiSq 

7.4940299  0.0062 
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Appendix  M,  NASA  High  Schedule  Growth  Linear  Regression  Model  Output 


Whole  Model 

Actual  by  Predicted  Plot  Residual  by  Predicted  Plot 


0.4-r 

0.3- 


ra  0.2 

D 

”  0  n 

0 

^  0.0 
|-0-H 
-0.2 


T - 1 - 1 - 1 - 1 - 1 - 1 - 

.8  1.0  1.2  1.4  1.6  1.8  2.0  2.2 

%  Schedule  Growth  Predicted 


JPL  Int'l  Develop 

Leverage  Plot  Leverage  Plot 


Summary  of  Fit 


RSquare  0.799249 

RSquare  Adj  0.732332 

Root  Mean  Square  Error  0.24891 

Mean  of  Response  1.161566 

Observations  (or  Sum  Wgts)  9 


Analysis  of  Variance 


Source 

DF  Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

2 

1 .4799994 

0.740000 

1 1 .9439 

Error 

6 

0.3717381 

0.061956 

Prob  >  F 

C.  Total 

8 

1.8517375 

0.0081 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Std  Beta 

Min  Value 

Max  Value 

Intercept 

0.899177 

0.101617 

8.85 

0.0001 

0 

JPL 

1.1877795 

0.268854 

4.42 

0.0045 

0.822943 

0 

1 

Int'l  Develop 

0.5868608 

0.203234 

2.89 

0.0278 

0.537883 

0 

1 
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Appendix  N,  NASA  High  Schedule  Growth  Linear  Regression  Model  Output:  Two 

Influential  Data  Points  Removed 

Whole  Model 

Regression  Plot  Actual  by  Predicted  Plot 


Residual  by  Predicted  Plot  JPL 

Leverage  Plot 


0.25-r 

0.20- 


16  0.15- 

I  0.10- 
0 

^  0.05- 

I  0.00- 

-0.05- 


-0.10- 


T - 1 - 1 - 1 - 1 - 1 - 1 - 

.8  1.0  1.2  1.4  1.6  1.8  2.0  2.2 

%  Schedule  Growth  Predicted 


Summary  of  Fit 


RSquare  0.908932 

RSquareAdj  0.890719 

Root  Mean  Square  Error  0.155666 

Mean  of  Response  1 .06886 

Observations  (or  Sum  Wgts)  7 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

1 

1 .2092745 

1.20927 

49.9042 

Error 

5 

0.1211595 

0.02423 

Prob  >  F 

C.  Total 

6 

1.3304340 

0.0009 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Std  Beta  Min  Value  Max  Value 

Intercept 

0.899177 

0.06355 

14.15 

<.0001 

0 

JPL 

1.1877795 

0.168138 

7.06 

0.0009 

0.953379  0  1 
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Appendix  O,  NASA  Low  Schedule  Growth  Linear  Regression  Model  Output 


Whole  Model 


Northrop  Grumman  Johns  Hopkins 

Leverage  Plot  Leverage  Plot 


Summary  of  Fit 


RSquare  0.276105 

RSquare  Adj  0.230861 

Root  Mean  Square  Error  0.181377 

Mean  of  Response  0.249582 

Observations  (or  Sum  Wgts)  35 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

2 

0.4015266 

0.200763 

6.1026 

Error 

32 

1 .0527272 

0.032898 

Prob  >  F 

C.  Total 

34 

1 .4542538 

0.0057 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Std  Beta 

Min  Value 

Max  Value 

Intercept 

0.252222 

0.033681 

7.49 

<.0001 

0 

Northrop  Grumman 

0.2429117 

0.110001 

2.21 

0.0345 

0.333603 

0 

1 

Johns  Hopkins 

-0.27371 

0.110001 

-2.49 

0.0182 

-0.3759 

0 

1 

124 


Appendix  P,  NASA  Low  Schedule  Growth  Linear  Regression  Model  Output 

Influential  Data  Point  Removed 


Whole  Model 


Actual  by  Predicted  Plot 


Northrop  Grumman 
Leverage  Plot 


-0.1  .0  .1  .2  .3  .4  .5  .6  .7  .8  .9  1.01.1 
Northrop  Grumman 
Leverage,  P=0.0022 


Johns  Hopkins 


-0.1  .0  .1  .2  .3  .4  .5  .6  .7  .8  .9  1.01.1 
Johns  Hopkins 
Leverage,  P=0.01 18 


Summary  of  Fit 


RSquare  0.389509 

RSquareAdj  0.350123 

Root  Mean  Square  Error  0.168731 

Mean  of  Response  0.252266 

Observations  (or  Sum  Wgts)  34 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Modei 

2 

0.5631070 

0.281554 

9.8894 

Error 

31 

0.8825755 

0.028470 

Prob  >  F 

C.  Totai 

33 

1 .4456826 

0.0005 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Std  Beta 

Min  Vaiue 

Max  Vaiue 

intercept 

0.252222 

0.031333 

8.05 

<.0001 

0 

Northrop  Grumman 

0.4113118 

0.123356 

3.33 

0.0022 

0.469337 

0 

1 

Johns  Hopkins 

-0.27371 

0.102332 

-2.67 

0.0118 

-0.37649 

0 

1 
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